Video semantic segmentation method, device, electronic device and storage medium

Abstract

The present disclosure provides a video semantic segmentation method, apparatus, electronic device and storage medium, the video semantic segmentation method comprising: obtaining first feature data corresponding to a video frame to be detected in video data, and historical feature data corresponding to historical video frames whose acquisition time is before the video frame to be detected in the video data; determining a first feature point that matches a position point of a complex image area in the video frame to be detected from a plurality of feature points corresponding to the first feature data; generating feature data of an enhanced feature point after semantic enhancement corresponding to the first feature point based on the historical feature data and the feature data of the first feature point; and determining target semantic information corresponding to each pixel point in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the plurality of feature points corresponding to the first feature data.

Description

[0001] Technology Neighborhood

[0002] The present disclosure relates to the field of deep learning technology, and more specifically, to a video semantic segmentation method, apparatus, electronic device, and storage medium. Background Art

[0003] Video semantic segmentation aims to assign a semantic label to each pixel in a video frame, so that the video frame can be segmented according to semantics. For example, different semantic objects such as pedestrians, bicycles, and animals in a video frame can be segmented to obtain semantic segmentation results.

[0004] Generally, when performing semantic segmentation on video data, semantic segmentation can be performed on each video frame in the video data to determine the semantic segmentation results for each video frame. The semantic segmentation results of each video frame can then be aligned, that is, the same object in different video frames of the video data is associated to obtain the semantic segmentation results corresponding to the video data. However, the above process of semantic segmentation of video data is relatively cumbersome, resulting in low semantic segmentation efficiency. Summary of the Invention

[0005] In view of this, the present disclosure at least provides a video semantic segmentation method, device, electronic device and storage medium.

[0006] In a first aspect, the present disclosure provides a video semantic segmentation method, comprising:

[0007] Acquire first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames in the video data whose acquisition time is before the video frame to be detected;

[0008] Determining, from a plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of a complex image region in the video frame to be detected; wherein the complex image region is a region including at least some pixel points of a plurality of target objects with different semantics;

[0009] generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point;

[0010] Target semantic information corresponding to each pixel in the to-be-detected video frame is determined based on the feature data of the enhanced feature point and feature data of other feature points except the first feature point among the plurality of feature points corresponding to the first feature data.

[0011] In the above method, after obtaining the first feature data of the video frame to be detected and the historical feature data of the historical video frames, based on the feature data of the first feature point in the first feature data and the historical feature data, the feature data of the enhanced feature point after semantic enhancement corresponding to the first feature point is generated, so that the feature data of the enhanced feature point includes the feature information of the video frame to be detected and the feature information of the historical video frames; then based on the feature data of the enhanced feature point and the feature data of other feature points, the target semantic information corresponding to each pixel point in the video frame to be detected is determined, and on the basis of using the historical feature data corresponding to the historical video frames in the video data, semantic segmentation with consistent time sequence between different video frames in the video data is achieved, thereby improving the efficiency of semantic segmentation.

[0012] At the same time, by determining the first feature point from multiple feature points corresponding to the first feature data, the first feature point is a feature point that matches the position point of the complex image area. Since the complex image area includes multiple target objects with different semantics, it is more difficult to determine the semantic information of the position point corresponding to the first feature point. Therefore, the first feature point can be semantically enhanced based on the historical feature data and the feature data of the first feature point to generate feature data of the enhanced feature point after the semantic enhancement corresponding to the first feature point. Subsequently, based on the feature data of the enhanced feature point and the feature data of other feature points, the target semantic information of each pixel point in the video frame to be detected can be determined more accurately, thereby improving the accuracy of semantic segmentation of the video frame to be detected.

[0013] In one possible implementation, determining, from the plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of the complex image area in the to-be-detected video frame includes:

[0014] Determining the adjacent similarity of each feature point corresponding to the first feature data; wherein the adjacent similarity is used to characterize the feature similarity distribution between the feature point and a plurality of surrounding feature points;

[0015] Based on the adjacent similarity, the first feature point is determined from a plurality of feature points corresponding to the first feature data.

[0016] Generally, compared with simple image areas including target objects with a single semantic meaning, complex image areas make a greater contribution to the accuracy and efficiency of semantic segmentation results. Therefore, in order to strike a balance between the accuracy and efficiency of semantic segmentation results, a first feature point that matches the position point of the complex image area in the video frame to be detected can be determined from multiple feature points corresponding to the first feature data. At the same time, considering that there will be large differences between the pixel information of target objects with different semantics, that is, there are differences in the pixel feature similarities of pixel points in the complex image area. Based on this, in the embodiment of the present disclosure, by determining the adjacent similarity of each feature point corresponding to the first feature data, and based on the adjacent similarity, the first feature point is more accurately determined from the multiple feature points corresponding to the first feature data.

[0017] In a possible implementation, determining the adjacent similarity of each feature point corresponding to the first feature data includes:

[0018] Taking each feature point corresponding to the first feature data as a current feature point, and determining a neighborhood feature matrix of the current feature point based on the position data of the current feature point and a preset neighborhood radius; wherein the neighborhood feature matrix includes feature vectors of each feature point located in the neighborhood of the current feature point;

[0019] The neighboring similarity corresponding to the current feature point is determined based on the neighborhood feature matrix and the feature vector of the current feature point.

[0020] By determining the neighborhood feature matrix of the current feature point, which includes feature information of other feature points located around the current feature point, the neighborhood similarity corresponding to the current feature point can be determined more accurately based on the neighborhood feature matrix and the feature vector of the current feature point, providing data support for the subsequent determination of the first feature point.

[0021] In a possible implementation, determining the neighboring similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point includes:

[0022] Determining at least one target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point; wherein the at least one target similarity includes: a first target similarity for characterizing the degree of similarity between the feature similarity distribution and a uniform distribution of each feature point in the neighborhood of the current feature point, and / or a second target similarity for characterizing the average feature similarity between each feature point in the neighborhood of the current feature point and the current feature point;

[0023] The adjacent similarity of the current feature point is determined based on the at least one target similarity.

[0024] Here, by setting at least one target similarity, the adjacent similarities of the current feature point can be determined more flexibly and accurately.

[0025] In one possible implementation, when the target similarity includes a first target similarity, determining the target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point includes:

[0026] Determine the feature similarity between the feature vector of each feature point in the neighborhood of the current feature point and the feature vector of the current feature point;

[0027] Based on the feature similarity, obtaining a similarity distribution vector corresponding to the current feature point;

[0028] Based on the similarity distribution vector and the determined uniform distribution vector, a first target similarity corresponding to the current feature point is determined.

[0029] In one possible implementation, when the target similarity includes the second target similarity, determining the second target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point includes:

[0030] Determine the cosine value of the angle between each eigenvector in the neighborhood feature matrix and the eigenvector of the current feature point;

[0031] The second target similarity corresponding to the target feature point is determined based on the cosine value of the angle corresponding to each eigenvector in the neighborhood feature matrix.

[0032] In a possible implementation, determining the first feature point from a plurality of feature points corresponding to the first feature data based on the adjacent similarity includes:

[0033] Determining the number of first feature points to be selected based on the number of feature points corresponding to the first feature data and a preset selection ratio;

[0034] Determine the selected number of first feature points from the multiple feature points corresponding to the first feature data in descending order of the adjacent similarities; and / or determine the first feature point from the multiple feature points corresponding to the first feature data based on the adjacent similarities and a set similarity threshold.

[0035] Here, by setting a plurality of selection methods, the first feature point can be determined more flexibly.

[0036] In one possible implementation, generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point includes:

[0037] determining a second feature point from a plurality of feature points corresponding to the historical feature data based on the position data of the first feature point and the radius of the area corresponding to the historical feature data;

[0038] Based on the feature data of the second feature point and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point is generated.

[0039] Since there is a temporal relationship between the historical video frame and the video frame to be detected, and the size of the target object in a moving state will change in different video frames, the second feature point can be determined more conveniently and efficiently from multiple feature points corresponding to the historical feature data based on the set area radius, the position data of the first feature point and the area radius corresponding to the historical feature data.

[0040] At the same time, because the semantic information of the second feature point is likely to be consistent with the semantic information of the first feature point, the feature data of the second feature point and the feature data of the first feature point can be used to more accurately enhance the semantics of the first feature point. Moreover, in the disclosed embodiment, compared with using the feature data of all feature points in the historical feature data to semantically enhance the first feature point, while ensuring the accuracy of semantic segmentation, the time complexity of semantic segmentation can be reduced, thereby improving the efficiency of semantic segmentation.

[0041] In one possible implementation, determining the second feature point from a plurality of feature points corresponding to the historical feature data based on the position data of the first feature point and the area radius corresponding to the historical feature data includes:

[0042] determining, from the historical feature data, an intermediate feature point that matches the position data of the first feature point;

[0043] Based on the area radius, determining a target area in the historical feature data with the intermediate feature point as the center;

[0044] Each feature point in the historical feature data that is located within the target area is determined as the second feature point.

[0045] In a possible implementation, the area radius corresponding to the historical feature data is determined according to the following steps:

[0046] Determining a candidate radius corresponding to the historical feature data based on a target number of frames corresponding to the historical feature data, and a set radius starting value, a frame number threshold, and an expansion coefficient;

[0047] If the candidate radius is smaller than a set radius cutoff value, determining the candidate radius as the area radius corresponding to the historical feature data;

[0048] In a case where the candidate radius is greater than or equal to the radius cutoff value, the radius cutoff value is determined as the area radius corresponding to the historical feature data.

[0049] Taking into account that the size of the target object in different video frames of the video data may change, the corresponding area radius may be determined for each historical feature data, and then the second feature point of each historical feature data may be determined more accurately based on the area radius.

[0050] In one possible implementation, generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point includes:

[0051] generating fused feature data based on the historical feature data and the feature data of the first feature point;

[0052] Performing feature extraction on the fused feature data to generate intermediate feature data;

[0053] Based on the intermediate feature data and the fused feature data, feature data of the semantically enhanced enhanced feature point corresponding to the first feature point is generated.

[0054] In one possible implementation, the target semantic information corresponding to each pixel in the video frame to be detected is obtained using a trained semantic segmentation neural network; the semantic segmentation neural network includes: a shared encoder, a feature point selection module, a temporal converter, and a segmentation decoder;

[0055] The shared encoder is used to perform feature extraction on the video frame to be detected and the historical video frame respectively, and obtain first feature data corresponding to the video frame to be detected and the historical video frame; the feature point selection module is used to determine the first feature point from multiple feature points corresponding to the first feature data;

[0056] The temporal converter is configured to perform semantic enhancement processing on the feature data of the first feature point based on the historical feature data corresponding to the historical video frame, to generate feature data of an enhanced feature point corresponding to the first feature point;

[0057] The segmentation decoder is used to determine the target semantic information corresponding to each pixel in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

[0058] In the above implementation, based on the video frames to be detected and the historical video frames, a semantic segmentation neural network is used to achieve semantic segmentation of different video frames in the video data with consistent time sequence, which improves the efficiency of semantic segmentation while ensuring the accuracy of semantic segmentation.

[0059] The description of the effects of the following devices, electronic equipment, etc. can be found in the description of the above method and will not be repeated here.

[0060] In a second aspect, the present disclosure provides a video semantic segmentation apparatus, comprising:

[0061] An acquisition module, configured to acquire first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames in the video data whose acquisition time is before the video frame to be detected;

[0062] A first determining module is configured to determine, from a plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of a complex image region in the video frame to be detected; wherein the complex image region is a region including at least some pixels of a plurality of target objects with different semantics;

[0063] A processing module, configured to generate feature data of a semantically enhanced feature point corresponding to the first feature point based on the historical feature data and the feature data of the first feature point;

[0064] The second determination module is used to determine the target semantic information corresponding to each pixel point in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

[0065] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the video semantic segmentation method described in the first aspect or any embodiment are performed.

[0066] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video semantic segmentation method as described in the first aspect or any one of the embodiments above are executed.

[0067] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of this specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A schematic diagram of a process for video semantic segmentation provided by an embodiment of the present disclosure is shown;

[0070] Figure 2 A schematic diagram illustrating a flow chart of a specific method for determining a first feature point in a video semantic segmentation method provided by an embodiment of the present disclosure is shown;

[0071] Figure 3 A schematic diagram of first feature data in a video semantic segmentation method provided by an embodiment of the present disclosure is shown;

[0072] Figure 4 A schematic diagram showing first feature data and historical feature data in a video semantic segmentation method provided by an embodiment of the present disclosure is shown;

[0073] Figure 5 A schematic diagram of the structure of a semantic segmentation neural network in a video semantic segmentation method provided by an embodiment of the present disclosure is shown;

[0074] Figure 6 A schematic diagram showing a flow chart of another video semantic segmentation method provided by an embodiment of the present disclosure;

[0075] Figure 7 A schematic diagram of the architecture of a video semantic segmentation device provided by an embodiment of the present disclosure is shown;

[0076] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0078] When performing semantic segmentation on video data, semantic segmentation can be performed on each video frame in the video data to determine the semantic segmentation results for each video frame. The semantic segmentation results for each video frame can then be aligned, i.e., the same object in different video frames of the video data can be associated to obtain the semantic segmentation results corresponding to the video data, thereby achieving time-consistent semantic segmentation for different video frames in the video data. However, the above process of obtaining the semantic segmentation results for the video data by associating the semantic segmentation results of each video frame is relatively cumbersome, and the efficiency of semantic segmentation is low.

[0079] Specifically, in order to achieve semantic segmentation with consistent timing between each video frame in the video data, the motion distortion between different video frames, such as optical flow, can be estimated to determine the semantic segmentation results of each video frame in the video data. For example, key video frames can be sampled from the video data, and the semantic segmentation results of the key video frames can be predicted using a semantic segmentation neural network. Then, the optical flow neural network can be used to determine the semantic segmentation results of other video frames in the video data except the key video frames based on the semantic segmentation results of the key video frames. However, in order to ensure the accuracy of the semantic segmentation of other video frames, the structure of the optical flow neural network tends to be complicated, which reduces the reasoning efficiency of the optical flow neural network, and thus causes the semantic segmentation efficiency of the video data to be low. At the same time, using multiple neural networks to perform semantic segmentation on each video frame in the video data makes the semantic segmentation process of the video data more cumbersome.

[0080] The disclosed embodiment provides a video semantic segmentation method, which, after obtaining first feature data of a video frame to be detected and historical feature data of historical video frames, generates feature data of a semantically enhanced feature point corresponding to the first feature point based on the feature data of the first feature point in the first feature data and the historical feature data, so that the feature data of the enhanced feature point includes feature information of the video frame to be detected and feature information of the historical video frames; then, based on the feature data of the enhanced feature point and the feature data of other feature points, determines the target semantic information corresponding to each pixel point in the video frame to be detected, and on the basis of using the historical feature data corresponding to the historical video frames in the video data, implements semantic segmentation with consistent time sequence between different video frames in the video data, thereby improving the efficiency of semantic segmentation.

[0081] At the same time, by determining the first feature point from multiple feature points corresponding to the first feature data, the first feature point is a feature point that matches the position point of the complex image area. Since the complex image area includes multiple target objects with different semantics, it is more difficult to determine the semantic information of the position point corresponding to the first feature point. Therefore, the first feature point can be semantically enhanced based on the historical feature data and the feature data of the first feature point to generate feature data of the enhanced feature point after the semantic enhancement corresponding to the first feature point. Subsequently, based on the feature data of the enhanced feature point and the feature data of other feature points, the target semantic information of each pixel point in the video frame to be detected can be determined more accurately, thereby improving the accuracy of semantic segmentation of the video frame to be detected.

[0082] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this disclosure for the above problems below should be the contributions made by the inventors to this disclosure during the disclosure process.

[0083] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition or explanation in subsequent drawings.

[0084] To facilitate understanding of the embodiments of the present disclosure, a video semantic segmentation method disclosed in the embodiments of the present disclosure is first introduced in detail. The execution subject of the video semantic segmentation method provided in the embodiments of the present disclosure can be a terminal device or a server, wherein the server can be, for example, a local server or a cloud server; the terminal device can be, for example, a mobile device, a personal digital assistant (PDA), a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the video semantic segmentation method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0085] See also Figure 1 FIG. 1 is a flow chart of a video semantic segmentation method provided by an embodiment of the present disclosure, wherein the method includes S101 to S104, wherein:

[0086] S101, obtaining first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames whose acquisition time is before the video frame to be detected in the video data;

[0087] S102, determining a first feature point that matches a position point of a complex image region in the video frame to be detected from a plurality of feature points corresponding to the first feature data; wherein the complex image region is a region including at least some pixels of a plurality of target objects with different semantics;

[0088] S103, generating feature data of a semantically enhanced feature point corresponding to the first feature point based on the historical feature data and the feature data of the first feature point;

[0089] S104 , determining target semantic information corresponding to each pixel in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

[0090] S101 to S104 are described in detail below.

[0091] For S101:

[0092] The historical video frames are video frames in the video data whose acquisition time is before the video frame to be detected, and the number of the historical video frames may be one or more frames.

[0093] During implementation, the target frame number corresponding to the video frame to be detected can be T+1, and the number of historical video frames can be T, that is, the first historical video frame (corresponding to the target frame number of 1), the second historical video frame, ..., the Tth historical video frame (corresponding to the target frame number of T) can be obtained. The value of T can be set as needed.

[0094] A video frame to be detected and at least one historical video frame can be obtained, and then feature extraction can be performed on the video frame to be detected to obtain first feature data corresponding to the video frame to be detected. Since feature extraction has been performed on the historical video frame when the historical video frame is used as the video frame to be detected, the historical feature data corresponding to the historical video frame exists. Therefore, the historical feature data corresponding to the historical video frame can be directly obtained without having to perform feature extraction on the historical video frame again, thereby avoiding the waste of resources caused by repeatedly extracting feature data of the historical video frame.

[0095] When there are multiple historical video frames, historical feature data corresponding to each historical video frame can be obtained.

[0096] For S102:

[0097] There are multiple feature points corresponding to the first feature data. For example, if the size of the first feature data is 56×56×128 (128 is the number of channels, and 56 is the corresponding length and width), then there are 56×56 feature points corresponding to the first feature data. For each feature point in the first feature data, according to the feature position of the feature point, the feature value matching the feature position is obtained from each channel to form the feature vector corresponding to the feature point. For example, for the feature point located in the 1st row and 1st column, the feature value located in the 1st row and 1st column is obtained from each channel to obtain the feature vector corresponding to the feature point in the 1st row and 1st column.

[0098] The first feature point can be determined from the multiple feature points corresponding to the first feature data, and other feature points other than the first feature point can also be obtained from the multiple feature points corresponding to the first feature data. The first feature point matches a position point on a complex image area in the video frame to be detected. The complex image area includes at least some pixel points of multiple target objects with different semantics. It can be seen that the complex image area can contain multiple semantic objects, or contain boundaries between different semantic objects.

[0099] In an optional embodiment, see Figure 2 As shown, determining a first feature point that matches a position point of a complex image area in a video frame to be detected from a plurality of feature points corresponding to the first feature data may include:

[0100] S201, determining the adjacent similarity of each feature point corresponding to the first feature data; wherein the adjacent similarity is used to characterize the feature similarity distribution between the feature point and multiple surrounding feature points;

[0101] S202 : Determine a first feature point from a plurality of feature points corresponding to the first feature data based on adjacent similarities.

[0102] Generally, compared with simple image regions including target objects with a single semantic meaning, complex image regions make a greater contribution to the accuracy and efficiency of semantic segmentation results. Therefore, in order to strike a balance between the accuracy and efficiency of semantic segmentation results, a first feature point that matches the position point of the complex image region in the video frame to be detected can be determined from multiple feature points corresponding to the first feature data. At the same time, considering that there will be large differences between the pixel information of target objects with different semantics, that is, there are differences in the pixel feature similarities of pixel points in the complex image region. Based on this, in the embodiment of the present disclosure, by determining the adjacent similarity of each feature point corresponding to the first feature data, and based on the adjacent similarity, the first feature point is more accurately determined from the multiple feature points corresponding to the first feature data.

[0103] For S201:

[0104] In an optional implementation, determining the adjacent similarity of each feature point corresponding to the first feature data may include step A1 and step A2, wherein:

[0105] Step A1: taking each feature point corresponding to the first feature data as a current feature point, and determining a neighborhood feature matrix of the current feature point based on the position data of the current feature point and a preset neighborhood radius; wherein the neighborhood feature matrix includes feature vectors of each feature point located in the neighborhood of the current feature point;

[0106] Step A2: Determine the neighboring similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point.

[0107] In step A1, the neighborhood radius can be determined based on multiple experiments. For example, the neighborhood radius r can be 1, 2, etc. Based on the position data of the current feature point and the pre-set neighborhood radius, the neighborhood feature matrix of the current feature point is determined. For example, if the position data of the current feature point in the first feature data is (u, v) and the neighborhood radius is r, then the neighborhood Q corresponding to the current feature point is n It is Q[ur:u+r][vr:v+r], and then based on the first feature data in the neighborhood Q n The feature vectors of each feature point in the neighborhood are used to generate the neighborhood feature matrix Q corresponding to the current feature point. n .

[0108] See also Figure 3 As shown, Figure 3 The first feature data 30 includes the current feature point 31. When the preset neighborhood radius is 1, each feature point within the rectangular frame 32 is a feature point located in the neighborhood. Each feature point corresponds to a feature vector. For example, if the first feature data includes 128 channels, the feature vector corresponding to each feature point includes 128 element values. The neighborhood feature matrix corresponding to the current feature point 31 can be generated based on the feature vectors corresponding to each feature point. Figure 3 It can be seen that the neighborhood feature matrix is ​​a 9×128 matrix.

[0109] In step A2, the neighborhood feature matrix and the feature vector of the current feature point can be used to determine the neighborhood similarity corresponding to the current feature point. The neighborhood similarity can be used to characterize the feature similarity distribution between the current feature point and multiple surrounding feature points (i.e., feature points other than the current feature point in the neighborhood).

[0110] In an optional implementation, determining the neighboring similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point may include:

[0111] Step A21: determining at least one target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point; wherein the at least one target similarity includes: a first target similarity for characterizing the degree of similarity between the feature similarity distribution and a uniform distribution of each feature point in the neighborhood of the current feature point, and / or a second target similarity for characterizing the average feature similarity between each feature point in the neighborhood of the current feature point and the current feature point;

[0112] Step A22: determining the adjacent similarity of the current feature point based on at least one target similarity.

[0113] During implementation, the neighborhood feature matrix and the feature vector of the current feature point may be used to determine at least one target similarity corresponding to the current feature point, wherein the at least one target similarity may include a first target similarity and / or a second target similarity.

[0114] The first target similarity is used to characterize the degree of similarity between the feature similarity distribution and the uniform distribution. The feature similarity distribution is the distribution of feature similarities between the current feature point and each feature point in its neighborhood. For example, if feature similarity distribution a is [0.1, 0.1, 0.7, 0.1], feature similarity distribution b is [0.2, 0.3, 0.25, 0.25], and uniform distribution is [0.25, 0.25, 0.25, 0.25], then the first target similarity is larger if feature similarity distribution a is less similar to the uniform distribution, while feature similarity distribution b is more similar to the uniform distribution and has a smaller first target similarity value.

[0115] The second target similarity is used to represent the average feature similarity between each feature point in the neighborhood of the current feature point and the current feature point.

[0116] When at least one target similarity includes a first target similarity, the first target similarity can be used as the neighboring similarity of the current feature point; when at least one target similarity includes a second target similarity, the second target similarity can be used as the neighboring similarity of the current feature point; when at least one target similarity includes the first target similarity and the second target similarity, the sum of the first target similarity and the second target similarity can be used as the neighboring similarity of the current feature point.

[0117] Furthermore, a Neighboring Similarity Matrix (NSM) matching the first feature data can be generated according to the position data of each feature point in the first feature data and the neighboring similarity corresponding to the feature point. The size of the Neighboring Similarity Matrix is ​​consistent with the first feature data.

[0118] Here, by setting at least one target similarity, the adjacent similarities of the current feature point can be determined more flexibly and accurately.

[0119] The process of determining the first target similarity is described below.

[0120] In an optional implementation, when the target similarity includes the first target similarity, determining the target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point may include steps B1 to B3, wherein:

[0121] Step B1, determining the feature similarity between the feature vector of each feature point in the neighborhood of the current feature point and the feature vector of the current feature point;

[0122] Step B2: Based on the feature similarity, obtain the similarity distribution vector corresponding to the current feature point;

[0123] Step B3: Determine the first target similarity corresponding to the current feature point based on the similarity distribution vector and the determined uniform distribution vector.

[0124] The feature similarity between each feature vector included in the neighborhood feature matrix and the feature vector of the current feature point can be determined, and the obtained feature similarities are used as element values ​​to form a similarity distribution vector corresponding to the current feature point.

[0125] During implementation, the similarity distribution vector P can be determined according to the following formula (1): sim :

[0126] P sim =SoftMax(Q n ·q T );(1)

[0127] Among them, q is the feature vector of the current feature point; Q n is the neighborhood feature matrix.

[0128] And the first target similarity D can be determined according to the following formula (2): KL :

[0129]

[0130] Among them, P u For uniform distribution, Psim is the similarity distribution vector, n b is the number of elements included in the similarity distribution vector.

[0131] Here, in P u Including n b elements, evenly distributed

[0132] Specifically, the quotient between the element value of each first element in the similarity distribution vector and the element value of the second element in the uniform distribution vector that matches the position of the first element can be determined; then the logarithm of the quotient corresponding to the first element is multiplied by the element value of the second element to obtain the product value corresponding to the first element; finally, the product values ​​corresponding to each first element in the similarity distribution vector are added to obtain the first target similarity corresponding to the target feature point.

[0133] The process of determining the second target similarity is described below.

[0134] In an optional embodiment, when the target similarity includes a second target similarity, determining the second target similarity corresponding to the current feature point based on the neighborhood feature matrix and the eigenvector of the current feature point may include: determining the cosine value of the angle between each eigenvector in the neighborhood feature matrix and the eigenvector of the current feature point; and determining the second target similarity corresponding to the target feature point based on the cosine values ​​of the angle corresponding to each eigenvector in the neighborhood feature matrix.

[0135] During implementation, the second target similarity D can be determined according to the following formula (3): cos :

[0136]

[0137] in, is the neighborhood feature matrix Q n The i-th eigenvector in n b is the number of elements included in the similarity distribution vector, and is also the neighborhood feature matrix Q n The number of eigenvectors included in .

[0138] For S202:

[0139] After obtaining the adjacent similarity of each feature point corresponding to the first feature data, the first feature point may be determined from the plurality of feature points corresponding to the first feature data according to the adjacent similarities corresponding to each feature point.

[0140] In an optional implementation, in S202, determining the first feature point from the plurality of feature points corresponding to the first feature data based on the adjacent similarity may include the following two methods:

[0141] Method 1: Determine the number of first feature points to be selected based on the number of feature points corresponding to the first feature data and the set selection ratio; determine the number of first feature points to be selected from the multiple feature points corresponding to the first feature data in descending order of adjacent similarities.

[0142] Method 2: Based on the adjacent similarity and a set similarity threshold, a first feature point is determined from a plurality of feature points corresponding to the first feature data.

[0143] In method 1, the selection ratio can be set as needed, for example, the selection ratio can be 40%, 50%, etc. If the number of feature points corresponding to the first feature data is 16×16 and the selection ratio is 50%, then the number of first feature points selected is determined to be 128. Then, 128 first feature points can be determined from the multiple feature points corresponding to the first feature data in descending order of adjacent similarities. That is, multiple target position points can be determined from the adjacent similarity matrix NSM matching the first feature data in descending order of adjacent similarities, and the feature point in the first feature data that matches the target position point is used as the first feature point.

[0144] In the second approach, the similarity threshold can be set as needed. From the plurality of feature points corresponding to the first feature data, feature points whose adjacent similarities are greater than or equal to the similarity threshold are selected as first feature points.

[0145] Here, by setting a plurality of selection methods, the first feature point can be determined more flexibly.

[0146] For S103 and S104:

[0147] Here, the feature data of all feature points included in the historical feature data can be used to semantically enhance the feature data of the first feature point, and generate feature data of the enhanced feature point after semantic enhancement corresponding to the first feature point. Among them, the feature data of the enhanced feature point includes the feature information in the historical feature data and the feature information of the first feature point, and the semantic information of the enhanced feature point is relatively rich. For example, the historical feature data and the feature data of the first feature point can be input into the Temporal Transformer, and the feature data of the first feature point can be semantically enhanced, so that the first feature point in the video frame to be detected can capture the temporal information and semantic information in the historical video frame, and generate feature data of the enhanced feature point with richer information, so that based on the feature data of the enhanced feature point, a semantic segmentation result with consistent temporal sequence corresponding to the video data can be obtained.

[0148] Alternatively, a second feature point that matches the position data of the first feature point can be selected from the historical feature data, and the feature data of the second feature point can be used to semantically enhance the feature data of the first feature point, thereby generating feature data of the semantically enhanced enhanced feature point corresponding to the first feature point. For example, the feature data of the second feature point and the feature data of the first feature point can be input into a temporal transformer, and the feature data of the first feature point can be semantically enhanced to generate feature data of the enhanced feature point.

[0149] In an optional embodiment, based on the historical feature data and the feature data of the first feature point, feature data of the semantically enhanced enhanced feature point corresponding to the first feature point is generated, including: based on the position data of the first feature point and the area radius corresponding to the historical feature data, determining the second feature point from multiple feature points corresponding to the historical feature data; based on the feature data of the second feature point and the feature data of the first feature point, generating feature data of the semantically enhanced enhanced feature point corresponding to the first feature point.

[0150] Since there is a temporal relationship between the historical video frame and the video frame to be detected, and the size of the target object in a moving state will change in different video frames, the second feature point can be determined more conveniently and efficiently from multiple feature points corresponding to the historical feature data based on the set area radius, the position data of the first feature point and the area radius corresponding to the historical feature data.

[0151] At the same time, since the semantic information of the second feature point is likely to be consistent with the semantic information of the first feature point, the feature data of the second feature point and the feature data of the first feature point can be used to more accurately enhance the semantics of the first feature point. Moreover, in the disclosed embodiment, compared with using the feature data of all feature points in the historical feature data to semantically enhance the first feature point, while ensuring the accuracy of semantic segmentation, the time complexity of semantic segmentation can be reduced, thereby improving the efficiency of semantic segmentation.

[0152] In the embodiment of the present disclosure, when there are multiple historical video frames, each historical video frame corresponds to a historical feature data, each historical feature data corresponds to an area radius, and different historical feature data correspond to different area radii. For example, the area radius corresponding to the historical feature data of the T-th historical video frame can be l T The area radius corresponding to the historical feature data of the T-1th frame of the historical video frame can be l T-1 .

[0153] For each frame of historical feature data, a second feature point on the historical feature data may be determined from a plurality of feature points corresponding to the historical feature data according to the area radius corresponding to the historical feature data and the position data of the first feature point.

[0154] In an optional implementation, the area radius corresponding to the historical feature data may be determined according to the following steps:

[0155] Step C1, determining a candidate radius corresponding to the historical feature data based on the target number of frames corresponding to the historical feature data, and the set radius starting value, frame number threshold, and expansion coefficient;

[0156] Step C2: if the candidate radius is smaller than the set radius cutoff value, the candidate radius is determined as the area radius corresponding to the historical feature data;

[0157] Step C3: When the candidate radius is greater than or equal to the radius cutoff value, the radius cutoff value is determined as the area radius corresponding to the historical feature data.

[0158] First, determine the candidate radius corresponding to the historical feature data. When the candidate radius is less than the radius cutoff value, determine the candidate radius as the area radius corresponding to the historical feature data; when the candidate radius is greater than or equal to the radius cutoff value, determine the radius cutoff value as the area radius corresponding to the historical feature data.

[0159] Taking into account that the size of the target object in different video frames of the video data may change, the corresponding area radius may be determined for each historical feature data, and then the second feature point of each historical feature data may be determined more accurately based on the area radius.

[0160] During implementation, the area radius l corresponding to the historical feature data of the tth frame can be determined according to the following formula (4): t :

[0161]

[0162] Where s is the starting radius value, ∈ is the expansion coefficient, t is the target number of frames, T is the frame threshold, and e is the radius cutoff value. s, ∈, and e can be set based on actual conditions. The frame threshold T is the number of historical video frames. t is the target number of historical video frames. For example, the target number of the T-th historical video frame is T (i.e., t = T), and the target number of the T-1th historical video frame is T-1 (i.e., t = T-1).

[0163] In an optional embodiment, based on the position data of the first feature point and the area radius corresponding to the historical feature data, determining the second feature point from multiple feature points corresponding to the historical feature data can include: determining an intermediate feature point that matches the position data of the first feature point from the historical feature data; based on the area radius, determining the target area in the historical feature data with the intermediate feature point as the center; and determining each feature point in the historical feature data located within the target area as the second feature point.

[0164] See also Figure 4 As shown, Figure 4 The first historical feature data 41 includes first feature data 41, first historical feature data 42, and second historical feature data 43. The first feature data 41 includes a first feature point 411. The first historical feature data 42 includes an intermediate feature point 421 that matches the position data of the first feature point 411. When the area radius corresponding to the first historical feature data 42 is 1, the target area in the first historical feature data can be obtained, that is, the area in the first rectangular frame 422 is the target area, and then each feature point in the first historical feature data located in the target area can be determined as the second feature point corresponding to the first historical feature data; the second historical feature data 43 includes an intermediate feature point 431 that matches the position data of the first feature point 411. When the area radius corresponding to the second historical feature data 43 is 2, the target area in the second historical feature data can be obtained, that is, the area in the second rectangular frame 432 is the target area, and then each feature point in the second historical feature data located in the target area can be determined as the second feature point corresponding to the second historical feature data.

[0165] In one possible implementation, based on the historical feature data and the feature data of the first feature point, feature data of the semantically enhanced enhanced feature point corresponding to the first feature point is generated, including: generating fused feature data based on the historical feature data and the feature data of the first feature point; performing feature extraction on the fused feature data to generate intermediate feature data; and generating feature data of the semantically enhanced enhanced feature point corresponding to the first feature point based on the intermediate feature data and the fused feature data.

[0166] Alternatively, fused feature data may be generated based on the feature data of the second feature point and the feature data of the first feature point; feature extraction may be performed on the fused feature data to generate intermediate feature data; and feature data of the semantically enhanced enhanced feature point corresponding to the first feature point may be generated based on the intermediate feature data and the fused feature data.

[0167] After obtaining the feature data of the enhanced feature point, feature extraction can be performed on the feature data of the enhanced feature point and the feature data of the other feature points, excluding the first feature point, among the multiple feature points corresponding to the first feature data, to determine the target semantic information corresponding to each pixel in the video frame to be detected, thereby obtaining a semantic segmentation result corresponding to the video frame to be detected. The semantic segmentation result may include a semantic segmentation map, in which each pixel in the semantic segmentation map corresponds to a semantic label, and different semantic labels may be marked with different colors.

[0168] In one possible implementation, the target semantic information corresponding to each pixel in the video frame to be detected is obtained using a trained semantic segmentation neural network; the semantic segmentation neural network includes: a shared encoder, a feature point selection module, a temporal converter, and a segmentation decoder;

[0169] The shared encoder is used to perform feature extraction on the video frame to be detected and the historical video frame respectively, and obtain the first feature data and the historical video frame corresponding to the video frame to be detected. The feature point selection module is used to determine the first feature point from the multiple feature points corresponding to the first feature data. The timing converter is used to perform semantic enhancement processing on the feature data of the first feature point based on the historical feature data corresponding to the historical video frame, and generate feature data of the enhanced feature point corresponding to the first feature point. The segmentation decoder is used to determine the target semantic information corresponding to each pixel point in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of the other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

[0170] In the above implementation, based on the video frames to be detected and the historical video frames, a semantic segmentation neural network is used to achieve semantic segmentation of different video frames in the video data with consistent time sequence, which improves the efficiency of semantic segmentation while ensuring the accuracy of semantic segmentation.

[0171] See also Figure 5 The structural diagram of the semantic segmentation neural network shown in Figure 5 This paper describes the process of video semantic segmentation method. Figure 6 As shown, the video semantic segmentation method may include:

[0172] S601: Obtain a video frame F to be detected in the video data. T+1 , and multiple frames of historical video frames. Among them, the multiple frames of historical video frames include historical video frames F T , historical video frame F T-1 , historical video frame F T-2 , historical video frame F T-3 ,…,historical video frame F1.

[0173] S602, using a shared encoder in a semantic segmentation neural network to detect the video frame F T+1 Perform feature extraction to obtain first feature data; and obtain historical feature data generated after the shared encoder extracts features from each historical video frame. That is, the historical feature data obtained includes: historical video frame F T Corresponding historical feature data, historical video frames F T-1 Corresponding historical feature data, ..., historical feature data corresponding to the historical video frame F1.

[0174] S603 : Determine a first feature point from a plurality of feature points corresponding to the first feature data through a feature point selection module in a semantic segmentation neural network.

[0175] During implementation, the adjacent similarity matrix NSM corresponding to the first feature data can be determined, wherein the determination process of NSM can refer to the above description and will not be described in detail here. Then, based on the NSM, the first feature point can be determined from the multiple feature points corresponding to the first feature data. Exemplarily, a selection ratio (such as 50%) can be set, and the first feature point is selected in the order of the adjacent similarities corresponding to each feature point in the first feature data indicated by the NSM from large to small. For example, the feature point corresponding to the gray box is the first feature point 51.

[0176] S604 , determining a second feature point from a plurality of feature points corresponding to the historical feature data based on the position data of the first feature point and the area radius corresponding to the historical feature data through a feature point selection module in the semantic segmentation neural network.

[0177] S605 : Input the feature data of the first feature point and the feature data of the second feature point into a temporal transformer to generate feature data of the semantically enhanced feature point corresponding to the first feature point.

[0178] During implementation, the feature data of the first feature point can be used as the query input of the time series converter; the feature data of the second feature point can be used as the key input of the time series converter; and the feature data of the second feature point can be used as the value input of the time series converter. It can be seen that the key input and the value input are the same. The multi-attention mechanism module in the time series converter performs feature fusion on the input data to generate first fused feature data; the first fused feature data and the feature data of the first feature point are then input into the feature processing layer Add&Norm in the time series converter for feature fusion to generate second fused feature data; the second fused feature data is then input into the Feed Forward Layer for feature extraction to generate third fused feature data; the third fused feature data and the second fused feature data are then input into the feature processing layer Add&Norm for feature fusion to generate feature data of the enhanced feature point corresponding to the first feature point after semantic enhancement.

[0179] S606. Using the segmentation decoder in the target neural network, the feature data of the enhanced feature point and the feature data of the other feature points except the first feature point among the multiple feature points corresponding to the first feature data are processed to determine the target semantic information corresponding to each pixel point in the video frame to be detected.

[0180] The video semantic segmentation method proposed in the embodiments of the present disclosure can be applied to scenarios requiring video semantic segmentation, such as autonomous driving, live broadcasting, and augmented reality (AR).

[0181] For example, when the video semantic segmentation method is applied to the field of autonomous driving, the video data may be road video data collected by a traveling device during driving. Based on the above-mentioned video semantic segmentation method, each road video frame in the road video data is semantically segmented to generate a semantic segmentation result corresponding to each road video frame; and then, based on the semantic segmentation result corresponding to each road video frame, the traveling device is controlled.

[0182] For example, the driving device may be an autonomous vehicle, a vehicle equipped with an Advanced Driving Assistance System (ADAS), or a robot, etc. When controlling the driving device, the driving device may be controlled to accelerate, decelerate, steer, brake, etc., or a voice prompt may be played to prompt the driver to control the driving device to accelerate, decelerate, steer, brake, etc.

[0183] By using the video semantic segmentation method to process each road video frame in the road video data, a semantic segmentation result corresponding to each road video frame is generated, thereby improving the accuracy and determination efficiency of the semantic segmentation result. Based on the semantic segmentation result corresponding to each road video frame, the driving device can be controlled more accurately and efficiently.

[0184] When the video semantic segmentation method is applied to an AR scene, the video data may be scene video data of a real-time scene of an AR device. The video semantic segmentation method proposed in the embodiment of the present disclosure is used to perform semantic segmentation on each scene video frame in the scene video data to generate a semantic segmentation result corresponding to each scene video frame. The matching target virtual object is then determined based on the semantic information of the target object indicated by the semantic segmentation result corresponding to each scene video frame, as well as the matching relationship between the pre-set semantics and the virtual object. The AR device is then controlled to display the scene video containing the target virtual object. For example, if the semantic information of the target object is a pedestrian, the matching target virtual object may be a pre-set virtual character; if the semantic information of the target object is a building, the matching target virtual object may be a pre-set virtual building, etc.

[0185] When the video semantic segmentation method is applied to a live broadcast scenario, the video data may be live video data. Using the video semantic segmentation method proposed in the embodiments of the present disclosure, semantic segmentation is performed on each live video frame in the live video data to generate a semantic segmentation result corresponding to each live video frame. Then, based on the semantic segmentation result corresponding to each live video frame, background replacement is performed on the live video frame. For example, pixel information of pixels in the live video frame that indicate semantic meanings other than humans, as indicated by the semantic segmentation result, may be replaced with a preset value to generate a live video frame with the background replaced.

[0186] Those skilled in the art will understand that in the above method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0187] Based on the same concept, the present disclosure also provides a video semantic segmentation device, see Figure 7 , which is a schematic diagram of the architecture of the video semantic segmentation apparatus provided by an embodiment of the present disclosure, includes an acquisition module 701, a first determination module 702, a processing module 703, and a second determination module 704. Specifically:

[0188] An acquisition module 701 is configured to acquire first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames in the video data whose acquisition time is before the video frame to be detected;

[0189] A first determining module 702 is configured to determine, from a plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of a complex image region in the video frame to be detected; wherein the complex image region is a region including at least some pixels of a plurality of target objects with different semantics;

[0190] A processing module 703 is configured to generate feature data of a semantically enhanced feature point corresponding to the first feature point based on the historical feature data and the feature data of the first feature point;

[0191] The second determination module 704 is used to determine the target semantic information corresponding to each pixel point in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

[0192] In one possible implementation, the first determining module 702, when determining, from the plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of the complex image area in the to-be-detected video frame, is configured to:

[0193] Determining the adjacent similarity of each feature point corresponding to the first feature data; wherein the adjacent similarity is used to characterize the feature similarity distribution between the feature point and a plurality of surrounding feature points;

[0194] Based on the adjacent similarity, the first feature point is determined from a plurality of feature points corresponding to the first feature data.

[0195] In a possible implementation, the first determining module 702, when determining the adjacent similarity of each feature point corresponding to the first feature data, is configured to:

[0196] Taking each feature point corresponding to the first feature data as a current feature point, and determining a neighborhood feature matrix of the current feature point based on the position data of the current feature point and a preset neighborhood radius; wherein the neighborhood feature matrix includes feature vectors of each feature point located in the neighborhood of the current feature point;

[0197] The neighboring similarity corresponding to the current feature point is determined based on the neighborhood feature matrix and the feature vector of the current feature point.

[0198] In a possible implementation, the first determining module 702, when determining the neighboring similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point, is configured to:

[0199] Determining at least one target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point; wherein the at least one target similarity includes: a first target similarity for characterizing the degree of similarity between the feature similarity distribution and a uniform distribution of each feature point in the neighborhood of the current feature point, and / or a second target similarity for characterizing the average feature similarity between each feature point in the neighborhood of the current feature point and the current feature point;

[0200] The adjacent similarity of the current feature point is determined based on the at least one target similarity.

[0201] In one possible implementation, when the target similarity includes a first target similarity, the first determining module 702, when determining the target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point, is configured to:

[0202] Determine the feature similarity between the feature vector of each feature point in the neighborhood of the current feature point and the feature vector of the current feature point;

[0203] Based on the feature similarity, obtaining a similarity distribution vector corresponding to the current feature point;

[0204] Based on the similarity distribution vector and the determined uniform distribution vector, a first target similarity corresponding to the current feature point is determined.

[0205] In one possible implementation, when the target similarity includes the second target similarity, the first determining module 702, when determining the second target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point, is configured to:

[0206] Determine the cosine value of the angle between each eigenvector in the neighborhood feature matrix and the eigenvector of the current feature point;

[0207] The second target similarity corresponding to the target feature point is determined based on the cosine value of the angle corresponding to each eigenvector in the neighborhood feature matrix.

[0208] In a possible implementation, the first determining module 702, when determining the first feature point from a plurality of feature points corresponding to the first feature data based on the adjacent similarity, is configured to:

[0209] Determining the number of first feature points to be selected based on the number of feature points corresponding to the first feature data and a preset selection ratio;

[0210] Determine the selected number of first feature points from the multiple feature points corresponding to the first feature data in descending order of the adjacent similarities; and / or determine the first feature point from the multiple feature points corresponding to the first feature data based on the adjacent similarities and a set similarity threshold.

[0211] In one possible implementation, the processing module 703, when generating feature data of a semantically enhanced feature point corresponding to the first feature point based on the historical feature data and the feature data of the first feature point, is configured to:

[0212] determining a second feature point from a plurality of feature points corresponding to the historical feature data based on the position data of the first feature point and the radius of the area corresponding to the historical feature data;

[0213] Based on the feature data of the second feature point and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point is generated.

[0214] In one possible implementation, the processing module 703, when determining the second feature point from the plurality of feature points corresponding to the historical feature data based on the position data of the first feature point and the area radius corresponding to the historical feature data, is configured to:

[0215] determining, from the historical feature data, an intermediate feature point that matches the position data of the first feature point;

[0216] Based on the area radius, determining a target area in the historical feature data with the intermediate feature point as the center;

[0217] Each feature point in the historical feature data that is located within the target area is determined as the second feature point.

[0218] In a possible implementation, the processing module 703 is configured to determine the area radius corresponding to the historical feature data according to the following steps:

[0219] Determining a candidate radius corresponding to the historical feature data based on a target number of frames corresponding to the historical feature data, and a set radius starting value, a frame number threshold, and an expansion coefficient;

[0220] If the candidate radius is smaller than a set radius cutoff value, determining the candidate radius as the area radius corresponding to the historical feature data;

[0221] In a case where the candidate radius is greater than or equal to the radius cutoff value, the radius cutoff value is determined as the area radius corresponding to the historical feature data.

[0222] In one possible implementation, the processing module 703, when generating feature data of a semantically enhanced feature point corresponding to the first feature point based on the historical feature data and the feature data of the first feature point, is configured to:

[0223] generating fused feature data based on the historical feature data and the feature data of the first feature point;

[0224] Performing feature extraction on the fused feature data to generate intermediate feature data;

[0225] Based on the intermediate feature data and the fused feature data, feature data of the semantically enhanced enhanced feature point corresponding to the first feature point is generated.

[0226] In one possible implementation, the target semantic information corresponding to each pixel in the video frame to be detected is obtained using a trained semantic segmentation neural network; the semantic segmentation neural network includes: a shared encoder, a feature point selection module, a temporal converter, and a segmentation decoder;

[0227] The shared encoder is used to perform feature extraction on the video frame to be detected and the historical video frame respectively, and obtain first feature data corresponding to the video frame to be detected and the historical video frame; the feature point selection module is used to determine the first feature point from multiple feature points corresponding to the first feature data;

[0228] The temporal converter is configured to perform semantic enhancement processing on the feature data of the first feature point based on the historical feature data corresponding to the historical video frame, to generate feature data of an enhanced feature point corresponding to the first feature point;

[0229] The segmentation decoder is used to determine the target semantic information corresponding to each pixel in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

[0230] In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0231] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 8802, and a bus 803. The memory 802 is used to store and execute instructions, and includes a memory 8021 and an external memory 8022. The memory 8021 is also referred to as internal memory, and is used to temporarily store the calculation data in the processor 801, as well as the data exchanged with the external memory 8022, such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the memory 8021. When the electronic device 800 is running, the processor 801 communicates with the memory 802 via the bus 803, so that the processor 801 executes the following instructions:

[0232] Acquire first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames in the video data whose acquisition time is before the video frame to be detected;

[0233] Determining, from a plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of a complex image region in the video frame to be detected; wherein the complex image region is a region including at least some pixel points of a plurality of target objects with different semantics;

[0234] generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point;

[0235] Target semantic information corresponding to each pixel in the to-be-detected video frame is determined based on the feature data of the enhanced feature point and feature data of other feature points except the first feature point among the plurality of feature points corresponding to the first feature data.

[0236] The specific processing flow of the processor 801 may refer to the description of the above method embodiment and will not be repeated here.

[0237] In addition, embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the video semantic segmentation method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0238] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the video semantic segmentation method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0239] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0240] The present disclosure relates to the field of augmented reality. By acquiring image information of a target object in a real environment, and then using various vision-related algorithms to detect or identify the relevant features, states, and attributes of the target object, an AR effect combining virtual and reality that matches the specific application is obtained.

[0241] Exemplarily, the target object may involve faces, limbs, gestures, movements, etc. related to the human body, or markers, landmarks related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, key point extraction and tracking of objects, object posture or depth detection, etc. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstruction, virtual effect overlay displays, etc. related to real scenes or objects, but may also involve special effects processing related to people, such as makeup beautification, body beautification, special effects display, virtual model display and other interactive scenes. Convolutional neural networks can be used to detect or identify the relevant features, states and attributes of the target object. The above-mentioned convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0243] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0244] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0245] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0246] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A video semantic segmentation method, characterized in that: include: Acquire first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames in the video data whose acquisition time is before the video frame to be detected; Determining, from a plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of a complex image region in the video frame to be detected; wherein the complex image region is a region including at least some pixel points of a plurality of target objects with different semantics; generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point; Target semantic information corresponding to each pixel in the to-be-detected video frame is determined based on the feature data of the enhanced feature point and feature data of other feature points except the first feature point among the plurality of feature points corresponding to the first feature data.

2. The method according to claim 1, characterized in that The step of determining, from the plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of the complex image area in the to-be-detected video frame, includes: Determining the adjacent similarity of each feature point corresponding to the first feature data; wherein the adjacent similarity is used to characterize the feature similarity distribution between the feature point and a plurality of surrounding feature points; Based on the adjacent similarity, the first feature point is determined from a plurality of feature points corresponding to the first feature data.

3. The method according to claim 2, characterized in that The determining of the adjacent similarity of each feature point corresponding to the first feature data includes: Taking each feature point corresponding to the first feature data as a current feature point, and determining a neighborhood feature matrix of the current feature point based on the position data of the current feature point and a preset neighborhood radius; wherein the neighborhood feature matrix includes feature vectors of each feature point located in the neighborhood of the current feature point; The neighboring similarity corresponding to the current feature point is determined based on the neighborhood feature matrix and the feature vector of the current feature point.

4. The method according to claim 3, characterized in that The determining the neighboring similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point includes: Determining at least one target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point; wherein the at least one target similarity includes: a first target similarity for characterizing the degree of similarity between the feature similarity distribution and a uniform distribution of each feature point in the neighborhood of the current feature point, and / or a second target similarity for characterizing the average feature similarity between each feature point in the neighborhood of the current feature point and the current feature point; The adjacent similarity of the current feature point is determined based on the at least one target similarity.

5. The method according to claim 4, characterized in that In a case where the target similarity includes a first target similarity, determining the target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point includes: Determine the feature similarity between the feature vector of each feature point in the neighborhood of the current feature point and the feature vector of the current feature point; Based on the feature similarity, obtaining a similarity distribution vector corresponding to the current feature point; Based on the similarity distribution vector and the determined uniform distribution vector, a first target similarity corresponding to the current feature point is determined.

6. The method according to claim 4, characterized in that In a case where the target similarity includes the second target similarity, determining the second target similarity corresponding to the current feature point based on the neighborhood feature matrix and the feature vector of the current feature point includes: Determine the cosine value of the angle between each eigenvector in the neighborhood feature matrix and the eigenvector of the current feature point; The second target similarity corresponding to the target feature point is determined based on the cosine value of the angle corresponding to each eigenvector in the neighborhood feature matrix.

7. The method according to claim 2, characterized in that The determining, based on the adjacent similarity, the first feature point from a plurality of feature points corresponding to the first feature data includes: Determining the number of first feature points to be selected based on the number of feature points corresponding to the first feature data and a preset selection ratio; Determine the selected number of first feature points from the multiple feature points corresponding to the first feature data in descending order of the adjacent similarities; and / or determine the first feature point from the multiple feature points corresponding to the first feature data based on the adjacent similarities and a set similarity threshold.

8. The method according to claim 1, characterized in that The step of generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point includes: determining a second feature point from a plurality of feature points corresponding to the historical feature data based on the position data of the first feature point and the radius of the area corresponding to the historical feature data; Based on the feature data of the second feature point and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point is generated.

9. The method according to claim 8, characterized in that The determining, based on the position data of the first feature point and the area radius corresponding to the historical feature data, of a second feature point from a plurality of feature points corresponding to the historical feature data includes: determining, from the historical feature data, an intermediate feature point that matches the position data of the first feature point; Based on the area radius, determining a target area in the historical feature data with the intermediate feature point as the center; Each feature point in the historical feature data that is located within the target area is determined as the second feature point.

10. The method according to claim 8, characterized in that Determine the area radius corresponding to the historical feature data according to the following steps: Determining a candidate radius corresponding to the historical feature data based on a target number of frames corresponding to the historical feature data, and a set radius starting value, a frame number threshold, and an expansion coefficient; If the candidate radius is smaller than a set radius cutoff value, determining the candidate radius as the area radius corresponding to the historical feature data; In a case where the candidate radius is greater than or equal to the radius cutoff value, the radius cutoff value is determined as the area radius corresponding to the historical feature data.

11. The method according to any one of claims 1 to 10, characterized in that: The step of generating, based on the historical feature data and the feature data of the first feature point, feature data of a semantically enhanced feature point corresponding to the first feature point includes: generating fused feature data based on the historical feature data and the feature data of the first feature point; Performing feature extraction on the fused feature data to generate intermediate feature data; Based on the intermediate feature data and the fused feature data, feature data of the semantically enhanced enhanced feature point corresponding to the first feature point is generated.

12. The method according to any one of claims 1 to 10, characterized in that: The target semantic information corresponding to each pixel in the video frame to be detected is obtained by using a trained semantic segmentation neural network; the semantic segmentation neural network includes: a shared encoder, a feature point selection module, a temporal converter, and a segmentation decoder; The shared encoder is used to perform feature extraction on the video frame to be detected and the historical video frame respectively, and obtain first feature data corresponding to the video frame to be detected and the historical video frame; the feature point selection module is used to determine the first feature point from multiple feature points corresponding to the first feature data; The temporal converter is configured to perform semantic enhancement processing on the feature data of the first feature point based on the historical feature data corresponding to the historical video frame, to generate feature data of an enhanced feature point corresponding to the first feature point; The segmentation decoder is used to determine the target semantic information corresponding to each pixel in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

13. A video semantic segmentation device, characterized in that: include: An acquisition module, configured to acquire first feature data corresponding to a video frame to be detected in the video data, and historical feature data corresponding to historical video frames in the video data whose acquisition time is before the video frame to be detected; A first determining module is configured to determine, from a plurality of feature points corresponding to the first feature data, a first feature point that matches a position point of a complex image region in the video frame to be detected; wherein the complex image region is a region including at least some pixels of a plurality of target objects with different semantics; A processing module, configured to generate feature data of a semantically enhanced feature point corresponding to the first feature point based on the historical feature data and the feature data of the first feature point; The second determination module is used to determine the target semantic information corresponding to each pixel point in the video frame to be detected based on the feature data of the enhanced feature point and the feature data of other feature points except the first feature point among the multiple feature points corresponding to the first feature data.

14. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the video semantic segmentation method as described in any one of claims 1 to 12 are performed.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the video semantic segmentation method according to any one of claims 1 to 12.

Images