Image information acquisition method and device, electronic equipment and chip

Abstract

The invention provides an image information acquisition method and device, electronic equipment and a chip, and relates to the technical field of digital image processing, in particular, the method comprises the following steps: firstly, carrying out superpixel segmentation on a to-be-processed pixel in a to-be-processed image to obtain superpixel information, the to-be-processed image comprising a reference frame image and a reference frame image, and based on a semi-global matching algorithm, dense motion vector information is generated according to the superpixel information, and the dense motion vector information is used for image fusion of the reference frame image and the reference frame image. According to the technical scheme, after the superpixel information is obtained from the to-be-processed image, the dense motion vector information serving as the image fusion information is further generated according to the superpixel information by using the two-dimensional semi-global matching algorithm, so that the obtaining speed of the image fusion information can be effectively improved, the operation complexity and the operation power consumption are reduced, and the image fusion efficiency is improved. Therefore, the image fusion efficiency can be effectively improved, and the requirement for equipment hardware is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of digital image processing, and in particular to an image information acquisition method, device, electronic device and chip. Background Art

[0002] Before fusing multiple frames of images, it is necessary to identify the same pixels of the base frame image and the reference frame image respectively, so as to achieve image fusion by using the same objects in different images.

[0003] However, when matching pixels of different images, in order to accurately fuse the pixels, the pixels need to be matched one by one, which greatly occupies the hardware memory used for image fusion and also prolongs the processing time of image fusion, affecting the user experience. Summary of the invention

[0004] In view of this, the present application provides an image information acquisition method, device, electronic device and chip, the main purpose of which is to quickly and accurately obtain fusion information between the baseline frame image and the reference frame image before image fusion.

[0005] In a first aspect, the present application provides a method for acquiring image information, comprising:

[0006] Performing super-pixel segmentation on pixels to be processed in an image to be processed to obtain super-pixel information, wherein the image to be processed includes a benchmark frame image and a reference frame image;

[0007] Based on a semi-global matching algorithm, dense motion vector information is generated according to the superpixel information, and the dense motion vector information is used for image fusion of the benchmark frame image and the reference frame image.

[0008] In a second aspect, the present application provides an image information acquisition device, comprising:

[0009] An acquisition module is used to perform super-pixel segmentation on the pixels to be processed in the image to be processed and obtain super-pixel information;

[0010] A generation module is used to generate dense motion vector information according to the superpixel information based on a two-dimensional semi-global matching algorithm, and the dense motion vector information is used as image fusion information.

[0011] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image information acquisition method described in the first aspect is implemented.

[0012] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the image information acquisition method described in the first aspect when executing the computer program.

[0013] In a fifth aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image information acquisition method described in the first aspect.

[0014] By means of the above technical solution, the present application provides an image information acquisition method, device, electronic device and chip. Compared with the existing pixel information of the base frame image and the reference frame image is obtained pixel by pixel to determine the image fusion information, the present application can use the image to be processed to perform superpixel division, obtain superpixel information, and use the superpixel information to achieve the acquisition of image fusion information. Specifically, the pixels to be processed in the image to be processed can be firstly segmented into superpixels to obtain superpixel information. The image to be processed includes the base frame image and the reference frame image. Then, based on the two-dimensional semi-global matching algorithm, dense motion vector information is generated according to the superpixel information. The dense motion vector information is used for the image fusion of the base frame image and the reference frame image. By applying the technical solution of the present application, after obtaining superpixel information from the image to be processed, a semi-global matching algorithm is further used to generate dense motion vector information as image fusion information based on the superpixel information. The superpixels of each part of the image can be directly recognized based on the superpixel information through the semi-global matching algorithm, and dense motion vector information can be generated based on the recognition results, thereby effectively improving the speed of acquiring image fusion information, reducing computational complexity and operating power consumption, and thereby effectively improving the efficiency of image fusion and reducing the requirements for device hardware.

[0015] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a process of obtaining image information provided by an embodiment of the present application is shown;

[0019] Figure 2 The overall flow chart of the SGM algorithm provided in the embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of 8-direction aggregation of the SGM algorithm of an embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram of a SLIC search in an embodiment of the present application is shown;

[0022] Figure 5 A flow chart of a method for superpixel updating provided in an embodiment of the present application is shown;

[0023] Figure 6 A schematic diagram showing the connection of modules of superpixel information provided by an embodiment of the present application is shown;

[0024] Figure 7 A schematic structural diagram of an image information acquisition device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0026] At present, the traditional method of obtaining the corresponding pixel information of the base frame image and the reference frame image pixel by pixel consumes a lot of computing power, which not only affects the efficiency of image fusion, but also has high requirements for device hardware. In view of the above problems, this embodiment provides a flow chart of an image information acquisition method, such as Figure 1 As shown, it can be applied to the end side of an image processing device (such as a smart phone, a tablet computer, a drone, a smart robot, and other smart terminals), and the method includes:

[0027] Step 101: perform superpixel segmentation on pixels to be processed in an image to be processed to obtain superpixel information. The image to be processed includes a benchmark frame image and a reference frame image.

[0028] In the field of computer vision, image segmentation refers to the process of subdividing a digital image into multiple image sub-regions (collections of pixels) (also called superpixels). A superpixel is a small region composed of a series of pixels that are adjacent in position and have similar characteristics such as color, brightness, and texture. Most of these small regions retain effective information for further image segmentation and generally do not destroy the boundary information of objects in the image. The result of image segmentation is a collection of sub-regions on the image (these sub-regions collectively cover the entire image) or a collection of contours extracted from the image (such as edge detection). Each pixel in a sub-region is similar in terms of a certain characteristic metric or calculated characteristic, such as color, brightness, and texture, and adjacent regions are very different in terms of a certain characteristic metric.

[0029] The content of this embodiment proposes to divide the pixels to be processed in the image to be processed into superpixels, and each superpixel has pixels to be processed with similar characteristics.

[0030] Step 102: Based on the semi-global matching (SGM) algorithm, dense motion vector information is generated according to the superpixel information, and the dense motion vector information is used as image fusion information.

[0031] After segmenting the superpixel information from the image to be processed, a semi-global matching (SGM) algorithm is used to accurately obtain dense motion vector information by using the recognition of superpixel information.

[0032] Semi-global matching (SGM) is a computer vision algorithm, such as Figure 2 The SGM algorithm shown is used to estimate dense disparity maps from rectified stereo image pairs. It has been widely adopted in real-time stereo vision applications such as robotics and advanced driver assistance systems due to its predictable runtime, favorable trade-off between result quality and computation time, and suitability for fast parallel implementation in ASIC or FPGA. The SGM algorithm concept is a data matching and cost aggregation algorithm based on a stereo matching algorithm for single-line image data. It can be described as a two-step approach: First, the cost of pixel correspondence is established for all possible difference labels in a defined search space. Second, the cost of these calculations is run along the scan line and regularized in the image domain using a cumulative dynamic programming scheme. The matching costs are accumulated and regularized and aggregated from multiple scan lines in different directions, and the best match is selected based on a winner-takes-all cost evaluation method.

[0033] Pixel-by-pixel stereo matching can be used to calculate disparity information in real time by measuring the degree of matching of each pixel between the two images. For a given set of stereo image pairs, for a pixel with coordinates (x, y), the set of pixels in the other image can be expressed as

[0034]

[0035] Where D is the maximum allowed parallax offset range, Indicates the possible corresponding coordinates of coordinate x in another image.

[0036] The search for the best match often produces multiple false matches. This problem can be solved by adding a regularization penalty term to avoid jittering matches between adjacent pixels, as described here:

[0037]

[0038] Where D(p, dp) is the similarity metric calculated pixel by pixel (at the position of disparity Dp), and R(p, dp, q, dq) is the regularization cost of pixels p and q at positions Dp and Dq, for all matching pairs N in the neighborhood. Such constraints can effectively improve the ability under dynamic programming, but this is still not a robust enough algorithm for disparity and depth calculation based only on intra-row information. A possible solution is to do it through global optimization, but here we will encounter the NP-Complete problem, which is still too complex for general real-time processing.

[0039] The core idea of ​​SGM is to complete the optimization in multiple directions in the dimension of row data and calculate the aggregate cost S(p,d), which is achieved by accumulating the matching cost of each pixel p at the position of disparity d. The number of directions will affect the speed of calculation. Usually, calculation with 16 directions can ensure better quality, while fewer directions can make the algorithm achieve faster operation speed. Typically, the implementation of 8 directions can complete the calculation in the forward and backward directions, such as Figure 3 The 8-way aggregation diagram of the SGM algorithm is shown, where the forward calculation is responsible for accumulating the matching costs of the left, upper left, above, and upper right; and the backward calculation is responsible for accumulating the matching costs of the right, lower right, below, and lower left.

[0040] The matching cost is composed of a matching term D(p,d) and a binary regularization term R(dp,dq). The former can be represented by any local image similarity metric, usually using the sum of absolute value differences or the square of the difference, or calculating the Hamming distance after Census transformation; while the regularization term has the following expression:

[0041]

[0042] Here, P1 and P2 are two outputs, where P1 < P2. Through a three-stage process, a small and uniform matching cost penalty term can be added to the disparity, which can generate smoother disparity information, such as for inclined surfaces; and the discontinuity of the disparity can be maintained at the disparity jump positions. To further maintain this discontinuity, the gradient information of the luminance component can also be added to the penalty term, because the discontinuity in the disparity usually corresponds to the discontinuity in the image luminance component:

[0043]

[0044] The accumulated matching cost S(p, d) can be expressed as:

[0045]

[0046] Here, by accumulating all Lr(p, d), all the matching costs for each pixel at disparity position d and along direction r are obtained. Each term can be expressed as:

[0047]

[0048] Here, the minimum cost min Lr(p - r, k) of the previous pixel is subtracted, which is for numerical stability because it is the same for all current pixels, so removing it will not affect the optimization result; the disparity of each pixel is expressed as d*(p) = argmind S(p, d), and the sub-pixel accuracy result can be obtained by curve fitting of d*(p). Considering that the image pairs from the two cameras are not actually processed completely symmetrically in the calculation, a consistency check can be performed by calculating the disparity in the opposite direction and discarding the disparity information with too large a difference between the first calculation and the reverse calculation. Further post-processing can optimize the texture region and fill the invalid regions.

[0049] Through the previous part, a one-dimensional SGM calculation can be constructed, that is, the accumulation process of semi-global matching in the case of binocular stereo matching applications, and it is assumed that SGM operates on the rectified image pairs. As mentioned above, this stereo correspondence problem is defined for a one-dimensional search range. However, for unconstrained optical flow estimation, the corresponding task is set in a two-dimensional search domain because any two-dimensional displacement is potentially possible. Next, an overview of how to embed the SGM concept into a coarse-to-fine method to achieve stable and efficient calculation of motion vectors (optical flow) will be given, and the key steps are as follows:

[0050] Step1. 2D semi-global matching

[0051] Motion vector search is an operation performed in 2D space. Here, it is extended from semi-global matching in 1D space to 2D space (SGM-2D) to perform this 2D search. Here, bijective discrete mapping is used to describe the transition from one-dimensional binocular disparity matching to two-dimensional optical flow search:

[0052]

[0053] The disparity signal originally on the 1D signal is converted into a motion vector φ(d) on the 2D image. The motion vector space Defined by a positive integer fm, fm specifies the maximum possible discrete motion vector:

[0054]

[0055] The offset (Δu, Δv) describes a pixel-accurate flow estimate, and its inverse mapping is:

[0056] φ -1 :O→D,withφ -1 (Δu, Δv) = d (9)

[0057] definition:

[0058]

[0059] and

[0060]

[0061] in is the L1 distance between two disparity values ​​in the motion vector domain O. Pκ is the penalty factor for scaling the slope of the linear function.

[0062] Step 2. Penalty Adjustment

[0063] As mentioned above, the matching cost in the one-dimensional binocular disparity calculation consists of the matching difference part and the regularization part:

[0064]

[0065] After mapping the matching cost to two-dimensional space:

[0066]

[0067] Next, consider the improvement of the regularization term and construct a regularization term of the following form:

[0068]

[0069] Step 3. Multi-scale motion estimation design Coarse-to-Fine Scheme for Mid-Scale Optical Flow

[0070] In order to improve the effectiveness of motion vector estimation, increase the reachable range of motion vectors and reduce computational overhead, the following methods and steps are used:

[0071] a) Generate a multi-layer image pyramid from the input image through a typical downsampling method

[0072] b) Estimation of the above motion vector at each level of the pyramid

[0073] c) The motion vector of the lower layer is enlarged according to the pyramid scaling ratio and then merged with the previous one

[0074] d) The input image is divided based on semantics or superpixels. As long as there is a valid motion vector at one position in the area, the entire area has a usable motion vector.

[0075] When this embodiment uses superpixel information to obtain image fusion information, compared with the currently existing methods, the superpixel information is a small area composed of a series of pixels with adjacent positions and similar characteristics such as color, brightness, and texture. The superpixel information contains multiple pixels to be processed, and there is a strong correlation between each pixel to be processed. The content of this embodiment matches the pixels to be processed with the SGM or SGM-2D algorithm, and there is no need to identify, match, and fuse the pixels to be processed one by one, which greatly reduces the calculation time and reduces the calculation complexity. By applying the technical solution of this embodiment, under the premise of combining the SGM2D algorithm with superpixel information, the calculation complexity and operating power consumption can be effectively reduced, thereby effectively improving the efficiency of image fusion and reducing the requirements for device hardware.

[0076] Further, as a refinement and extension of the above embodiment, as a possible embodiment, based on a two-dimensional semi-global matching algorithm, dense motion vector information is generated according to superpixel information, including:

[0077] A feature extraction is performed on a first pixel to be processed in the image to be processed to obtain a first feature, and the first pixel to be processed is located in a first superpixel to be processed; a feature extraction is performed on a second pixel to be processed in the image to be processed to obtain a second feature, and the second pixel to be processed is located in a second superpixel to be processed, and the second superpixel to be processed is an adjacent superpixel of the first superpixel to be processed; a matching score between the first feature and the second feature is calculated based on the superpixel information, and the matching score is used to represent the correlation between the first pixel to be processed and the second pixel to be processed; and a cost aggregation is performed on the matching score to generate dense motion vector information.

[0078] In the calculation process of the two-dimensional semi-global matching algorithm, it is necessary to first extract features from the pixels to be processed in the image data, extract the first feature for the first pixel to be processed, and extract the second feature for the second pixel to be processed, wherein the second superpixel to be processed is the adjacent superpixel of the first superpixel to be processed. Exemplarily, the extracted feature can be a Census feature. After the feature is extracted, the first feature and the second feature are used to perform matching calculations, the matching score between the first feature and the second feature is calculated, and the correlation between the first feature and the second feature is determined according to the matching score. The higher the correlation, the greater the possibility that the objects in the image corresponding to the first feature and the second feature are the same object, and the matching score can be used to further calculate the dense motion vector information. The scheme of this embodiment proposes to extract the features of superpixel information and compare the superpixel information from the pixels to be processed in the image to be processed to determine the correlation of pixels in the image. Compared with pixel-by-pixel extraction and matching, the content of this embodiment directly compares the correlation between pixels, increases the semantic recognition between correlations, can effectively improve the acquisition speed of image fusion information, reduce the computational complexity and running power consumption, and thus effectively improve the efficiency of image fusion, and reduce the requirements for device hardware.

[0079] As a refinement and extension of the above embodiment, in a possible embodiment, calculating a matching score between the first feature and the second feature according to the superpixel information includes:

[0080] Confirm the first superpixel information corresponding to the first feature; confirm the multiple second superpixel information corresponding to the multiple second features; and calculate multiple matching scores between the first superpixel information and the multiple second superpixel information based on the Hamming distance.

[0081] Regarding the specific calculation method of the matching score, after determining the first superpixel information corresponding to the first feature and the multiple second superpixel information corresponding to the second feature, the matching score between the first superpixel information and the second superpixel information can be calculated by calculating the Hamming distance. In the process of comparing the first feature with the second feature, for the first feature, there may be multiple second features corresponding to the first feature, each second feature corresponds to a group of superpixel information, and the matching score between the first feature and each second feature is calculated.

[0082] The technical solution of this embodiment proposes a specific calculation method for calculating the matching score between the first feature and the second feature: Hamming distance match calculation. At the same time, it is further explained that the association between the first feature and the second feature is not necessarily a one-to-one relationship, but may also be a one-to-many relationship, and the first feature and the second feature need to be matched one by one. Exemplarily, the Censue feature corresponding to the first pixel to be processed current_pixel is Census_cur, and there may be multiple Census features of the second pixel to be processed in its adjacent domain. When the number is three, they are represented by Candidiate_Censue_P1, Candidiate_Censue_P2 and Candidiate_Censue_P3 respectively; these three features are represented by superpixel labels, which can be Candidate_SP_Lable1, Candidate_SP_Lable2 and Candidate_SP_Lable3, and are represented by superpixel centers, which can be Candidate_SP_Center1, Candidate_SP_Center2 and Candidate_SP_Center3, respectively, where the superpixel label label and the superpixel center SP_center are both superpixel information, and the superpixel label can be a symbol representing the superpixel. The process of calculating the matching scores separately can be expressed as:

[0083] Candidate_Score1=match_sp(Census_cur,Pixel_value_cur,pixel_pos_cur,Candidate_Censue_P1,Candidate_SP_Lable1,Candidate_SP_Center1)

[0084] Candidate_Score2=match_sp(Census_cur,Pixel_value_cur,pixel_pos_cur,Candidate_Censue_P2,Candidate_SP_Lable2,Candidate_SP_Center2)

[0085] Candidate_Score3=match_sp(Census_cur,Pixel_value_cur,pixel_pos_cur,Candidate_Censue_P3,Candidate_SP_Lable3,Candidate_SP_Center3)

[0086] Among them, Candidate_Score is the matching score, match_sp is the Hamming distance calculation of the superpixel information, Pixel_value_cur is the color position of the first pixel to be processed, and pixel_pos_cur is the spatial position of the first pixel to be processed.

[0087] By adopting the steps of this embodiment, the calculation of each matching score between the first superpixel information and multiple second superpixel information can be quickly realized. Because the first feature and the second feature are respectively in adjacent superpixel domains, the matching relationship between the first superpixel information and the second superpixel information can be identified, so that the correlation between the first feature and the second feature in the image can be confirmed, and the semantic recognition of the image can be realized, effectively improving the fusion accuracy of the image fusion information.

[0088] In a possible embodiment, before performing cost aggregation on the matching scores to generate dense motion vector information, the method further includes:

[0089] Among multiple matching scores, confirm the matching scores to be verified whose scores are lower than the preset scores and the matching result scores whose scores are higher than the preset scores; extract the target pixel in the first superpixel information to be verified corresponding to the matching score to be verified; extract the superpixel center of the second superpixel information to be verified corresponding to the matching score to be verified, and obtain the superpixel center to be verified; confirm the matching result of the superpixel information of the target pixel and the superpixel center to be verified; based on the matching result, make a first adjustment to the matching score between the first superpixel information to be verified and the second superpixel information to be verified; and use the first adjusted matching score as the matching result score;

[0090] Cost aggregation is performed on the matching scores to generate dense motion vector information, including:

[0091] Cost aggregation is performed on the matching result scores to generate dense motion vector information.

[0092] In this embodiment, in order to accurately obtain the matching score, this embodiment further proposes the relevant steps of performing the first adjustment on the matching score. In the process of comparing the first feature with the second feature, the multiple matching scores generated may be higher than the preset score or lower than the preset score. If the matching score is higher than the preset score, it means that they are very similar in semantics and there is a high probability that they belong to the same object. They should have more similar image features. For the domain pixels with the same superpixel label, the original Hamming distance match calculation operation can be maintained during matching.

[0093] If the current pixel and the neighboring pixel to be matched have different superpixel information, it means that they do not have a very strong similarity and may belong to different objects. However, the reason why it is not directly reduced to confirm that they do not belong to the same object is that the two superpixels may belong to the same object, but due to the efficiency limitation of the superpixel hardware calculation, the separate target pixel falls into different superpixels. Therefore, it is necessary to confirm based on the similarity with the superpixel center, that is, to further verify the target pixel and the second superpixel information corresponding to the verified matching score, and judge whether it is necessary to further adjust the matching score based on the confirmation result.

[0094] The technical solution of this embodiment proposes different technical solutions for different comparison results of the first feature and the second feature, and further confirms the size relationship between the matching result and the preset score to realize the semantic recognition of different super-pixel information in the image to be processed, and further explains the problem that pixels with the same semantics are not in the same super-pixel due to the super-pixel size limitation. At the same time, the recognition of pixels with the same semantics can further realize the super-pixel internal pixel anomaly caused by the influence of noise on the base frame image and the reference frame image. By adjusting the matching score, the same object in the image can be accurately identified, and whether the pixel value in the super-pixel is affected by noise can be further identified, so that the accuracy of the matching score is further guaranteed.

[0095] In a possible embodiment, confirming the superpixel information matching result between the target pixel and the center of the superpixel to be verified includes:

[0096] Calculate the first absolute value of the color distance difference between the target pixel and the center of the superpixel to be verified and the second absolute value of the spatial distance difference; obtain the absolute sum of the first absolute value and the second absolute value; and confirm the correspondence between the target pixel and the center of the superpixel to be verified based on the absolute sum.

[0097] Regarding the specific adjustment method of the matching score, this embodiment proposes to compare the spatial distance difference of pixels with the pixel center of the superpixel for pixel values ​​that do not fall into the superpixel, and judge whether the semantic content of the superpixel and the target pixel is the same by comprehensively comparing the color space distance and position space distance of the pixel center of the superpixel of the target pixel value, and further adjust the matching score between the first superpixel information and the second superpixel information corresponding to the target pixel, so that subsequent image fusion can achieve image fusion more intelligently.

[0098] Exemplarily, the calculation of the target pixel and the superpixel center SP_neighbor to be verified can be defined as:

[0099] Distance_Neighbor_SP_Center=Color_Diff(Pixel_value_cur,Candidate_SP_Center.color)+Pos_Diff(pixel_pos_cur,Candidate_SP_Center.pos);

[0100] Here, Distance_Neighbor_SP_Center represents the absolute sum value, Color_Diff(Pixel_value_cur,Candidate_SP_Center.color) is used to calculate the absolute value of the color distance difference between the target pixel and the superpixel center to be verified corresponding to the RGB three channels, and Pos_Diff(pixel_pos_cur,Candidate_SP_Center.pos) is used to calculate the positional spatial distance between the target pixel and the second superpixel center, which is the sum of the absolute values ​​of the x-direction difference and the y-direction difference.

[0101] The color distance difference of the RGB three channels is as follows:

[0102] abs(R_cur–R_SP_Center)+abs(G_cur–G_SP_Center)+abs(B_cur–B_SP_Center)

[0103] abs(R_cur–R_SP_Center) is the color distance difference of the R channel, abs(G_cur–G_SP_Center) is the color distance difference of the G channel, and abs(B_cur–B_SP_Center) is the color distance difference of the B channel;

[0104] The position space distance is as follows:

[0105] abs(pixel_pos_cur.x-Candidate_SP_Center.pos.x)+abs((pixel_pos_cur.y-Candidate_SP_Center.pos.y)

[0106] Among them, abs(pixel_pos_cur.x-Candidate_SP_Center.pos.x) represents the difference in the x direction, and abs((pixel_pos_cur.y-Candidate_SP_Center.pos.y) represents the difference in the y direction.

[0107] In a possible embodiment, first adjusting the matching score between the first superpixel information to be verified and the second superpixel information to be verified based on the matching result includes:

[0108] If the target pixel and the second superpixel center correspond to each other, obtain an absolute sum value; adjust a first adjustment parameter between the first superpixel information and the second superpixel information according to the absolute sum value; and make a first adjustment to the matching score between the first superpixel information to be verified and the second superpixel information to be verified according to the first adjustment parameter.

[0109] When it is determined that there is a mutual correspondence between the target pixel and the second superpixel center, it is confirmed that the target pixel and the second superpixel center belong to the same object in the image. Therefore, it is necessary to adjust the matching scores of the first superpixel to be verified and the second superpixel to be verified to which the target pixel belongs, wherein the adjustment value is determined by the semantic correlation between the target pixel and the second superpixel center.

[0110] For example, Distance_Neighbor_SP_Center is used to determine how similar the neighboring points with different superpixel labels from the current pixel are to the current pixel in terms of semantic content, and accordingly, the parameter adjustment information Weight_score is generated:

[0111] Weight_score_center=Weight_score_LUT(Distance_Neighbor_SP_Center)

[0112] Among them, Weight_score_LUT is a lookup table predefined as a parameter. When the input Distance_Neighbor_SP_Center is smaller, the parameter adjustment information output is closer to 1. When the input Distance_Neighbor_SP_Center is larger, the output is a value greater than 1 and less than MAX_WEIGHT_SCORE, where MAX_WEIGHT_SCORE is a predefined value. The larger the value, the more obvious the influence of the superpixel on the matching score, and finally the first adjustment parameter Weight_score_center is output.

[0113] Apply Weight_score to the matching score:

[0114] Candidate_Score_center_1=Censue_score1*Weight_score_center_1;

[0115] Censue_score1 is the matching score before adjustment, and Candidate_Score_center_1 is the matching score after adjustment. In a possible embodiment, cost aggregation is performed on the matching scores to generate dense motion vector information, including:

[0116] A plurality of third superpixels to be processed in the base frame image and a plurality of fourth superpixels to be processed in the reference frame image are obtained, wherein the matching scores between the plurality of third superpixels to be processed and the plurality of fourth superpixels to be processed are the highest; a first edge intersection between the plurality of third superpixels to be processed is determined, and a second edge intersection between the plurality of fourth superpixels to be processed is determined; a distance difference between the first edge intersection and the second edge intersection is calculated; a second adjustment is made to the matching score according to the distance difference; and cost aggregation is performed on the matching score after the second adjustment to generate dense motion vector information.

[0117] After implementing SGM2D to calculate the matching score between the first feature and the second feature, the motion vectors Motion_vector_x and Motion_vector_y in the x and y directions are generated. After completing the division of superpixels and confirming the pixels inside superpixels, at the edge of the superpixel, that is, the intersection of two superpixel information, the superpixel segmentation at this position is usually not very accurate, and more processing is needed here to improve the accuracy of the edge position.

[0118] Therefore, in this embodiment, it is proposed to extract the first edge intersection point from the plurality of third superpixels to be processed in the base frame image, and extract the second edge intersection point from the reference frame image, so that the plurality of superpixels constituting the two edge intersection points correspond to each other, and the matching score value between them is the highest. The first edge intersection point represents the edge information of the superpixel in the base frame image, and the second edge intersection point represents the edge information of the superpixel in the reference frame image. After the accuracy between the internal pixels of the superpixels in the two images is respectively confirmed by using the semantic relationship of the pixels, the edge information of the superpixel is further used to represent the position abnormality of the superpixel, and by comparing the spatial positions of the first edge intersection point and the second edge intersection point, the superpixel edges of the base frame image and the reference frame image are matched and adjusted, so that the base frame image and the reference frame image will not be difficult to achieve image fusion due to the error between the edge intersection points, thereby ensuring the accuracy of the image fusion information finally generated.

[0119] In a possible embodiment, calculating the distance difference between the first edge intersection point and the second edge intersection point includes:

[0120] Perform feature extraction on the first edge intersection to obtain third feature information; perform feature extraction on the second edge intersection to obtain fourth feature information; calculate the absolute value of the distance difference between the operators in the third feature information and the fourth feature information; confirm the sum of the absolute values ​​of the distance differences of multiple operators, and use the sum of the absolute values ​​as the distance difference.

[0121] This embodiment discloses a specific calculation method for the distance difference between the first edge intersection and the second edge intersection. Exemplarily, the HoG features are extracted from the first edge intersection and the second edge intersection respectively, and the distance difference between the first edge intersection and the second edge intersection is expressed by calculating the distance difference of each operator when the two intersections are simultaneously in the HoG feature, so as to realize the confirmation of the distance difference between the first edge intersection and the second edge intersection. By adopting the technical solution of this embodiment, the distance difference between the first edge intersection and the second edge intersection can be quantified, so as to realize the adjustment of the distance between the first edge intersection and the second edge intersection, so as to meet the adjustment of the pixel edge position of the base frame image and the reference frame image, so as to ensure the accuracy of the fusion of the base frame image and the reference frame image.

[0122] Exemplarily, assuming that the intersection of multiple third superpixels A, B and C to be processed in the reference frame image at the edge position is Current_intersection_point, and in the reference frame, there are also fourth superpixels D, E, F to be processed, and the intersection of their edge positions is Base_intersection_point. Because the matching score between superpixels A, B and C and superpixels D, E, F is the highest, this point is the intersection of the superpixel edges closest to Current_intersection_point in the Euclidean distance, then the HoG features are first extracted respectively:

[0123] Current_intersection_point_hog=HoG(Current_intersection_area);

[0124] Base_intersection_point_hog=HoG(Base_intersection_area);

[0125] Among them, Current_intersection_area and Base_intersection_area are image information of a small neighborhood extracted based on the positions of Current_intersection_point and Base_intersection_point in order to calculate the HoG feature. The small neighborhood can be the superpixels A, B and C or superpixels D, E, F mentioned above, and the HoG feature Current_intersection_point_hog of the first edge intersection and the HoG feature Base_intersection_point_hog of the second edge intersection are output.

[0126] It should be noted that the HoG feature can be replaced by other traditional feature operators, and no detailed restrictions are made here.

[0127] After obtaining the feature description of the intersection position between superpixel edges, it is necessary to define the distance metric between features to determine the similarity between features; the distance difference Distance_intersection_pt of the feature is defined as:

[0128] Distance_intersection_pt=D(Current_intersection_point_hog,Base_intersection_point_hog)

[0129] Where D is the distance metric between the results of two feature operators. The distance metric can be defined as the L1 distance between the two feature operators, such as

[0130] Current_intersection_point_hog-Base_intersection_point_hog;

[0131] It should be noted that the definition here is not limited to L1 distance, and it can also be the distance metric of other traditional feature operators, which will not be repeated here.

[0132] In a possible embodiment, adjusting the matching score according to the distance difference includes:

[0133] Obtain superpixel matching scores between multiple third superpixels to be processed and multiple fourth superpixels to be processed; determine a second adjustment parameter based on the superpixel matching score and the distance difference; and adjust the matching score using the second adjustment parameter.

[0134] When calculating the adjustment parameters based on the superpixel intersection matching score, the parameter adjustment information is obtained according to the matching score Distance_intersection_pt and the lookup table Weight_score_LUT of the preset parameters of the superpixel edge intersection:

[0135] Weight_score_border=Weight_score_LUT(Distance_intersection_pt),

[0136] Among them, Weight_score_LUT is a lookup table predefined as a parameter. When the input Distance_intersection_pt is smaller, the output weight is closer to 1. When the input Distance_intersection_pt is larger, the output is a value greater than 1 and less than MAX_WEIGHT_SCORE, where MAX_WEIGHT_SCORE is a predefined value. The larger the value, the more obvious the influence of the superpixel on the matching score, and finally the second adjustment parameter Weight_score_ is output.

[0137] Apply Weight_score to the matching score:

[0138] Candidate_Score_border_2=Censue_score2*Weight_score_border_2;

[0139] Censue_score2 is the matching score before adjustment, Candidate_Score_center_2 is the matching score after adjustment, and Weight_score_center_1 is the second adjustment parameter.

[0140] In a possible embodiment, before adjusting the matching score by using the second adjustment parameter, the method further includes:

[0141] Calculate adjustment factors of the first adjustment parameter and the second adjustment parameter respectively; calculate a third adjustment parameter based on the adjustment factors and the first adjustment parameter and the second adjustment parameter;

[0142] The matching score is adjusted using the second adjustment parameter, including:

[0143] The matching score is adjusted using a third adjustment parameter.

[0144] After respectively obtaining the first adjustment parameter and the second adjustment parameter, wherein the first adjustment parameter mainly realizes the matching of the pixel value inside the superpixel, and the second adjustment parameter mainly realizes the matching of the edge of the superpixel. In this embodiment, the third adjustment parameter is calculated by combining the first adjustment parameter and the second adjustment parameter to jointly realize the overall matching score calculation of the superpixels in the image. In this embodiment, it is proposed to comprehensively consider the first adjustment parameter and the second adjustment parameter, rather than considering the first adjustment parameter or the second adjustment parameter alone, so that the third adjustment parameter finally generated can more accurately reflect the accurate adjustment parameter for image fusion of the base frame image and the reference frame image.

[0145] For example, in terms of adjustment, the first adjustment of the superpixel center and the second adjustment of the edge information may be correspondingly fused:

[0146] weight_score3_final=alpha*weight_score_center_1+(1-alpha)*weight_score_border_3

[0147] Among them, weight_score3_final is the third adjustment parameter, weight_score_center_1 is the first adjustment parameter, weight_score_border_3 is the second adjustment parameter, and alpha is the adjustment factor.

[0148] The matching scores are calculated for the corresponding fusion of the benchmark frame image and the reference frame image according to the matching degree inside the superpixel and the matching degree at the edge of the superpixel, which can greatly speed up the accuracy of subsequent image fusion and reduce the fusion waiting time.

[0149] In a possible embodiment, performing superpixel segmentation on pixels to be processed in an image to be processed to obtain superpixel information includes:

[0150] The simple linear iterative clustering (SLIC) algorithm is used to perform superpixel segmentation on the pixels to be processed in the processed image to obtain superpixel information.

[0151] Simple linear iterative clustering (SLIC) is an easy-to-implement algorithm that converts a color image into a 5-dimensional feature vector in LAB color space and XY coordinates, then constructs a distance metric for the 5-dimensional feature vector and performs local clustering of image pixels.

[0152] The SLIC calculation process is as follows:

[0153] 1. Initialize seed points (cluster centers): Distribute seed points evenly in the image according to the set number of superpixels. Assuming that the image has a total of N pixels, which are pre-divided into K superpixels of the same size, then the size of each superpixel is N / K, and the distance (step length) between adjacent seed points is approximately S = sqrt(N / K);

[0154] 2. Reselect the seed point in the n*n neighborhood of the seed point (usually n=3). The specific method is: calculate the gradient values ​​of all pixels in the neighborhood, and move the seed point to the place with the smallest gradient in the neighborhood. The purpose of this is to avoid the seed point falling on the contour boundary with a large gradient, so as not to affect the subsequent clustering effect;

[0155] 3. Assign a class label to each pixel in the neighborhood around each seed point (i.e., which cluster center it belongs to). Unlike the standard k-means search in the entire image, SLIC limits the search range to 2S*2S, which can accelerate the convergence of the algorithm, as shown in the figure below. Note that the expected superpixel size is S*S, but the search range is 2S*2S, such as Figure 4 As shown, Figure 4 A search schematic diagram of SLIC is provided.

[0156] 4. Distance measurement. Including color distance and spatial distance. For each searched pixel, calculate the distance between it and the seed point. The distance calculation method is as follows:

[0157]

[0158] Where dc represents the color distance, ds represents the spatial distance, and Ns is the maximum spatial distance within the class, which is defined as Ns = S = sqrt(N / K) and is applicable to each cluster. The maximum color distance Nc varies with different images and clusters, so a fixed constant m (value range [1,40], generally 10) is used instead. The final distance metric D' is as follows:

[0159]

[0160] Since each pixel point will be searched by multiple seed points, each pixel point will have a distance to the surrounding seed points, and the seed point corresponding to the minimum value is taken as the cluster center of the pixel point;

[0161] 5. Iterative optimization. In theory, the above steps are iterated continuously until the error converges (which can be understood as the cluster center of each pixel point no longer changes). In practice, it is found that 10 iterations can achieve a relatively ideal effect for most images, so the number of iterations is generally 10;

[0162] 6. Enhance connectivity. After the above iterative optimization, the following defects may occur: multiple connections, superpixels are too small, a single superpixel is cut into multiple discontinuous superpixels, etc. These problems can be solved by enhancing connectivity. The main idea is: create a new label table, all elements in the table are -1, and reallocate discontinuous superpixels and superpixels that are too small to adjacent superpixels in a "Z"-shaped direction (from left to right, from top to bottom), and assign the traversed pixels to the corresponding labels until all points are traversed.

[0163] This embodiment includes proposing a specific algorithm to implement the step of segmenting the pixels to be processed of the image to be processed, and its main advantages are:

[0164] 1) The generated superpixels are as compact and neat as cells, and the neighborhood features are easier to express. In this way, the pixel-based method can be easily transformed into a superpixel-based method;

[0165] 2) It can not only segment color images, but also grayscale images;

[0166] 3) There are very few parameters that need to be set. By default, you only need to set the number of pre-segmented superpixels;

[0167] 4) Compared with other superpixel segmentation methods, SLIC is more ideal in terms of running speed, compactness of generated superpixels, and contour preservation.

[0168] In a possible embodiment, a simple linear iterative clustering algorithm (SLIC) is used to perform superpixel segmentation on the pixels to be processed in the image to be processed to obtain superpixel information, including:

[0169] The image to be processed is converted into color space to generate the color space coordinates of the pixel to be processed; in the color space coordinates, a simple linear iterative clustering algorithm is used to determine the color space coordinates of the superpixel center of the pixel to be processed.

[0170] In this embodiment, specific steps of using SLIC are proposed, and after the color space coordinates are established, the specific application of SLIC is realized.

[0171] Exemplarily, the color space coordinate establishment step may include:

[0172] In superpixel calculation, the distance in color space is calculated by converting RGB image information into LAB space. Therefore, it is necessary to convert the color space based on the input RGB information and the RGB to LAB color space conversion method. The specific conversion method is:

[0173] (1) Convert sRGB to XYZ space

[0174] The input RGB data is normalized to 8 bits, the relationship between the channel data and the threshold value 0.04045 is determined, and it is determined whether to use linear transformation of r=R / 12.92 or exponential processing: =pow((R+0.055) / 1.055,2.4). The same processing is applied to each RGB channel. According to the different weights of the XYZ three channels, the RGB data processed in the previous step are weighted to obtain the XYZ data.

[0175] (2) Convert XYZ to LAB space

[0176] According to the CIE standard, set the epsilon and kappa values, set the white point 3D data Xr / Yr / Zr, use Xr / Yr / Zr to normalize the input XYZ value, and determine whether to use the exponential formula or the linear formula by judging the relationship between the normalized data Xr / Yr / Zr and the threshold epsilon, and calculate fx / fy / fz. The required Lab space 3D data lval / aval / bval is calculated through fx / fy / fz.

[0177] After determining the color space coordinates, a simple linear iterative clustering algorithm is used to determine the color space coordinates of the superpixel center of the pixel to be processed, including:

[0178] 1. Initialize the center position of each superpixel

[0179] The main function of this part GetLABXYSeeds_ForGivenStepSize is to take k seed values ​​as uniform spatial pixel samples;

[0180] Its input is STEP information, and optional information edgemag and flag perturbseeds; its output is seed coordinate information: kseedsl / kseedsa / kseedsb / kseedsx / kseedsy;

[0181] (1) First, calculate the horizontal and vertical stripe information: xstripes / ystripes according to the input STEP information and image size information;

[0182] (2) Calculate the remainder xerr / yerr of the image size if it is not divisible, and adjust the xtrips / ystrips information;

[0183] (3) Calculate the ratio of xerr / yerr in image strips;

[0184] (4) Determine the number of seeds based on the total number of image stripes and initialize kseedsl / kseedsa / kseedsb / kseedsx / kseedsy;

[0185] (5) Loop for each stripes and calculate the kseedsl / kseedsa / kseedsb / kseedsx / kseedsy information; store the lab information of the seed position in kseedsl / kseedsa / kseedsb, and store the seed coordinates in kseedsx / kseedsy

[0186] 2. Calculate and update the superpixel center position of the current position pixel by pixel, mainly as follows Figure 5 As shown, Figure 5 A flow chart of a method for superpixel updating is disclosed, which is completed by the steps of updating the selection of the superpixel neighborhood of the current pixel, calculating the minimum color space distance, and confirming whether to update the superpixel center position.

[0187] In this section, we mainly introduce superpixel calculation based on pixel dimension iteration. This section is mainly divided into the following parts: loop optimization for each superpixel subset, calculation of color space distance, updating superpixel subset using pixels in the minimum color space, and updating superpixel information. The main difference between this part and traditional superpixel calculation is that it is a superpixel calculation based on local pixel dimension iteration. For the entire superpixel calculation process, each pixel only needs to be read from the external DDR storage and once. Compared with the traditional superpixel calculation that requires multiple pixel values ​​to be obtained from the external DDR, it can effectively reduce DDR bandwidth information and facilitate hardware optimization and implementation deployment. Let's take a look at the key steps one by one:

[0188] 2.1. Loop optimization for each superpixel subset

[0189] First, let's clarify the processing flow on the architecture. Here we introduce a new method for calculating SLIC superpixels, which is more efficient than the traditional SLIC superpixel calculation with multiple frame-level iterations. In each iteration, a subset of image pixels is used to calculate the superpixel center, which allows each superpixel center to be updated with a small fraction of the distance calculation and memory bandwidth, while still accurately calculating the position of the superpixel center;

[0190] Choosing a suitable subset sampling strategy is the basis for ensuring the convergence of the iterative algorithm. The hardware architecture from the pixel perspective is mainly considered. By dividing the image pixels into subsets of equal size, different subsets are used to update the superpixels in each iteration. The subsets are traversed in a loop to ensure that all image pixels are taken into account. Compared with the traditional superpixel calculation method in which superpixels are divided into subsets of equal size and different subsets are used to update superpixels in each iteration, that is, the superpixel calculation method from the pixel perspective, the superpixel subsets are traversed in a loop order. The hardware architecture based on the pixel perspective requires less data for external storage DRAM than the traditional solution, and is more friendly to the hardware computer vision algorithm architecture that requires pipeline processing.

[0191] 2.2 Calculating color space distance

[0192] In the pixel-by-pixel iteration process, the color space distance needs to be calculated first. This distance is used to prepare for the subsequent judgment of the superpixel relationship between the current pixel and the surrounding 9 neighborhoods based on the color space distance. The calculation method of the color space distance mainly considers two aspects, one is the color distance and the other is the space distance. The color distance can be the distance in the CIELAB space or the distance between the original RGB information. The traditional superpixel color distance is calculated using the distance in the CIELAB color space. In this case, in order to save the amount of calculation, the distance calculation based on the RGB color space can be used optionally.

[0193] Here, the distance in color space is defined as the sum of the distance in RGB color space between the current pixel and the superpixel center and the spatial distance, as defined by Ds:

[0194]

[0195] The RGB color space is defined as the absolute value of the difference between the three RGB channels:

[0196] d rgb =|(r k -r i )|+|(g k -g i )|+|(b k -b i )| (18)

[0198] The spatial distance is simplified to the absolute value and interpolation of the horizontal and vertical distances.

[0199] d xy =|x k -x i )|+|(y k -y i )| (19)

[0201] Here, a LAB space-based metric is optionally used, which relies on the weighted LAB space distance and the Euclidean distance between pixels to obtain the final distance metric Ds:

[0202]

[0203] Where dlab is the color distance in the lab space, which is calculated as the square root of the sum of the squares of the differences between the three channels l&a&b:

[0204]

[0205] The measure of spatial distance is Euclidean distance, which is the square root of the sum of the squares of the differences in the horizontal and vertical directions;

[0206]

[0207] In practice, the first method, i.e., color and space measurement based on RGB space, can greatly reduce the hardware calculation complexity while keeping the effect basically the same;

[0208] 2.3 Superpixel Core Computation

[0209] In the superpixel generation architecture, each module is composed as follows Figure 6 As shown, Figure 6 A schematic diagram of the module connection of superpixel information is disclosed, in which the core modules are as follows: image data buffer module, superpixel storage module, superpixel calculation module, calculation initialization module, and superpixel information output module. The core design ideas and calculation processes of these modules are introduced below:

[0210] 2.3.1 Image Data Buffer Module

[0211] The main function of the image data buffer module is to buffer the image sequence of line scanning or sequential reading from DDR according to the needs of superpixel calculation. Here, the buffering is performed in units of lines, and the number of lines required is mainly determined by the size of the superpixel. Image buffering occupies a considerable part of the total cost of hardware overhead and requires detailed design. The image buffer module can obtain the input RGB image information and output it after buffering for several cycles. The size of the image, the interval S of the superpixels and the number of superpixels are the key determinants of the size of this buffer module. In this solution, considering that there are S*S size pixel blocks around the center of each superpixel, in the initialization stage, the superpixel center is actually set at the center of the image block S*S. In the subsequent stage, the center position and value will become the mean of all pixel values ​​related to the superpixel center. Therefore, the centers of the 9 superpixels must be prepared before the new pixels arrive.

[0212] In the initialization phase, since the superpixel center is specified by the initialization value, there is no need to wait until all relevant pixels are acquired before knowing it. Therefore, the minimum delay occurs at this time, which is also the phase that consumes the least storage unit: MinimumDelay = Image_Width*S + (Image_Width / 2)*S

[0213] For other stages of the non-initialization phase, since it is necessary to wait for all pixels related to the current superpixel center to arrive before knowing the updated superpixel center, the delay is: TypicalDelay = 3*Image_Width*S

[0214] When multiple iterations are required, the data buffering needs to be reconsidered. Two methods are designed here to support more than one superpixel calculation. Taking 2 iterations as an example, there are two methods to obtain the results of 2 iterations:

[0215] Method 1: Keep the number of existing data buffers unchanged and reduce the actual superpixel size to half of the original size. In this way, the row buffers required in the row direction are half of the original size. The saved row buffers are used for data buffering to support the second superpixel iterative calculation.

[0216] Method 2: Increase the number of buffers to twice the current number, so that twice as much input data can be buffered. After the first iteration is completed, continue to buffer the corresponding data buffer in the 3*3 superpixel area of ​​the first iteration, and use the superpixel center obtained in the first iteration for the second iteration;

[0217] 2.3.2 Superpixel Storage Module

[0218] Superpixel information storage module, which is used to store superpixel center information. Each superpixel information storage unit is composed of several superpixel row storages, and each superpixel row storage is responsible for storing the superpixel information of the corresponding row, that is, the column coordinate information SuperPixelCol of the superpixel; in the superpixel calculation process, the superpixel information storage unit is responsible for receiving the update of the superpixel information and writing it into the storage unit; after the current stage ends, the superpixel information storage unit needs to start supporting the superpixel calculation process of the next stage after a certain delay time. In this process, information representing a superpixel will be read, which is the mean of all pixels associated with the superpixel center; when the input is RGB information, the storage is also stored according to the three channels of RGB, and also includes row and column coordinate information;

[0219] 2.3.3 Superpixel Calculation Module

[0220] The superpixel calculation unit is actually the core module in the entire superpixel processing and calculation process. The core function of this module is to compare each input pixel pixel_input with its 9 surrounding superpixels to obtain the information of which of the 9 adjacent superpixels the pixel_input falls into, and synchronize the corresponding superpixel information that needs to be updated;

[0221] In this process, we first need to understand that the superpixel information of the current pixel 9 neighborhood is stored through a 3*3 storage structure. This 3*3 storage structure will be updated when a new superpixel center is passed in. The update method will be described later.

[0222] When a new pixel pixel_in arrives, it is necessary to calculate the color distance Drgb and the spatial distance Dxy respectively according to the previous color and spatial distance metrics, and combine them to calculate Ds; the LAB color space can be optionally used for calculation here.

[0223] After obtaining the color and distance metrics of the input pixel pixe_in and the surrounding 9 superpixel centers, the minimum value of the color and distance metrics and its corresponding superpixel center index are obtained by sorting;

[0224] Regarding the update method of the 3*3 storage structure, the typical method is that when a new superpixel center enters, each column in the 3*3 storage structure moves one column to the left, and the new superpixel center is loaded into the third column; when a new pixel value is entered, it is necessary to calculate the color and spatial distance of the current pixel relative to the surrounding 9 superpixel centers; there are several cases for the update of the 3*3 storage structure:

[0225] One case is that during the initialization process, when there are not enough superpixel centers entering, all superpixel centers are assigned initialization values. When a new superpixel center enters, the existing superpixel center moves one column to the left, and the new superpixel center is filled to the last column.

[0226] Init Value Init Value Init Value Init Value Init Value SuperPixel(0,0) Init Value Init Value SuperPixel(1,0)

[0227] Updated to:

[0228] Init Value Init Value Init Value Init Value SuperPixel(0,0) SuperPixel(0,1) Init Value SuperPixel(1,0) SuperPixel(1,1)

[0229] One case is a loop for each row. In a row, it continues until the last pixel in the row enters. When a new pixel located in a new superpixel enters, each row of superpixel centers in the current 3*3 storage is moved up by one row, and the newly entered superpixel center is filled in the position of the third row;

[0230] Init Value Init Value SuperPixel(0,0) SuperPixel(0,n-1) SuperPixel(0,n) SuperPixel(1,0) SuperPixel(1,n-1) SuperPixel(1,n) SuperPixel(2,0)

[0231] Updated to:

[0232] Init Value SuperPixel(0,0) SuperPixel(0,1) SuperPixel(0,n) SuperPixel(1,0) SuperPixel(1,1) SuperPixel(1,n) SuperPixel(2,0) SuperPixel(2,1)

[0233] In terms of the number of iterations, considering the cost of hardware implementation, it needs to be constrained to 1-3 times; the constraints here come from the data pipeline and hardware complexity; from the perspective of data pipeline, considering that the superpixel module in the system is actually on a pipeline that requires real-time processing at the pixel level, such an architecture is beneficial to the overall system to reduce the reading of external DDR storage, and the constraint on the algorithm module is that the algorithm also needs to use a limited pixel buffer for processing, and it needs to output the superpixel label information of each pixel at the same rate as the input pixel; therefore, offline processing cannot be used here, that is, it cannot be in the form of loading a block of data, and then iterating multiple times and then outputting the superpixel label information of each pixel; the implementation of multiple iterations here needs to take into account the calculation process of the pipeline. When it is necessary to iterate 2-3 times, in fact, after one round of calculation, the superpixel information is updated, and it is necessary to continue to reuse the data in the row buffer in the existing field of view, and combine the first superpixel information as the initial state of the second iteration, and then iterate again, and update the superpixel information, and finally output the superpixel label information of each pixel;

[0234] 2.3.4 Calculation initialization module

[0235] The superpixel calculation initialization module is responsible for completing the initialization process of the information required for superpixel calculation. The initialization process can be regarded as a simple version of the superpixel storage unit, which does not include accumulation, averaging and division operations. The initialization stage is responsible for initializing the corresponding center pixel value within the S*S pixel range to ensure that the relevant calculation process of the superpixel calculation unit can be performed after the two rows of superpixel centers begin to enter;

[0236] 2.3.5 Superpixel Information Output Module

[0237] The superpixel information output module is the last module in the pipeline. The row and column coordinate information obtained by the previous superpixel calculation module is used to label each output pixel with a superpixel label. The labeling method is as follows:

[0238] OutputSuperPixelID=SuperPixelsPerrow*SuperPixelRow+SuperPixelCol

[0239] Where SuperPixelsPerrow is the number of superpixels in each row, and SuperPixelRow and SuperPixelCol are the row and column address information of each superpixel;

[0240] You can choose to output superpixels according to pixels, or you can choose to output superpixel labels according to superpixels. If it is the latter, the superpixel information output module can be replaced by the superpixel storage module.

[0241] 3. Post-processing of superpixel calculation

[0242] This can be done by using filtering modules such as median filtering to post-process superpixel information.

[0243] Further, as Figure 1 The specific implementation of the method shown in the embodiment provides an image information acquisition device, such as Figure 7 As shown, the device includes: an acquisition module 71 and a generation module 72.

[0244] An acquisition module 71 is used to perform super-pixel segmentation on pixels to be processed in an image to be processed to obtain super-pixel information, wherein the image to be processed includes a benchmark frame image and a reference frame image;

[0245] The generating module 72 is used to generate dense motion vector information according to the superpixel information based on a semi-global matching algorithm, and the dense motion vector information is used for image fusion of the benchmark frame image and the reference frame image.

[0246] In a specific application scenario, the generating module 72 is specifically used for:

[0247] Extract features from the image to be processed to obtain a first feature;

[0248] Extract features from the image to be processed to obtain a second feature;

[0249] Calculating a matching score between the first feature and the second feature according to the superpixel information, wherein the matching score is used to represent a correlation between the first pixel to be processed and the second pixel to be processed;

[0250] Cost aggregation is performed on the matching scores to generate dense motion vector information.

[0251] In a specific application scenario, the generating module 72 is specifically used for:

[0252] Confirming first superpixel information corresponding to the first feature;

[0253] Confirming a plurality of second superpixel information corresponding to a plurality of the second features;

[0254] Based on the Hamming distance, multiple matching scores between the first superpixel information and multiple second superpixel information are calculated.

[0255] In a specific application scenario, the generating module 72 is further specifically used for:

[0256] Among the multiple matching scores, confirm the matching scores to be verified whose scores are lower than the preset scores and the matching result scores whose scores are higher than the preset scores;

[0257] Extracting target pixels from the first superpixel information to be verified corresponding to the matching score to be verified;

[0258] Extracting superpixel centers of the second superpixel information to be verified corresponding to the matching score to be verified to obtain the second superpixel center;

[0259] The target pixel is a pixel to be processed whose matching score with the center of the second superpixel exceeds a preset score;

[0260] Confirming the correspondence between the target pixel and the second superpixel center;

[0261] Performing a first adjustment on a matching score between the first superpixel information to be verified corresponding to the second superpixel center and the target second superpixel information to be verified;

[0262] Using the first adjusted matching score as the matching result score;

[0263] The generating module 72 is specifically used to perform cost aggregation on the matching result scores to generate dense motion vector information.

[0264] In a specific application scenario, the generating module 72 is specifically used for:

[0265] Calculate a first absolute value of a color distance difference between the target pixel and a second superpixel center and a second absolute value of a spatial distance difference;

[0266] Obtaining an absolute sum of the first absolute value and the second absolute value;

[0267] The correspondence between the target pixel and the second superpixel center is confirmed according to the absolute sum value.

[0268] In a specific application scenario, the generating module 72 is specifically used for:

[0269] If the target pixel and the second superpixel center correspond to each other, obtaining the absolute sum value;

[0270] Adjust a first adjustment parameter between the first superpixel information and the second superpixel information according to the absolute sum value;

[0271] According to the first adjustment parameter, a first adjustment is performed on the matching score between the first superpixel information to be verified and the second superpixel information to be verified.

[0272] In a specific application scenario, the generating module 72 is specifically used for:

[0273] Obtaining a plurality of third superpixels to be processed in the benchmark frame image and a plurality of fourth superpixels to be processed in the reference frame image, wherein the plurality of third superpixels to be processed and the plurality of fourth superpixels to be processed have the highest mutual matching scores;

[0274] Determine a first edge intersection point between the plurality of third superpixels to be processed, and determine a second edge intersection point between the plurality of fourth superpixels to be processed;

[0275] Calculating a distance difference between the first edge intersection point and the second edge intersection point;

[0276] Performing a second adjustment on the matching score according to the distance difference;

[0277] Cost aggregation is performed on the matching score adjusted for the second time to generate dense motion vector information.

[0278] In a specific application scenario, the generating module 72 is specifically used for:

[0279] Performing feature extraction on the first edge intersection to obtain third feature information;

[0280] Performing feature extraction on the second edge intersection to obtain fourth feature information;

[0281] Calculating the absolute value of the distance difference between the operators in the third feature information and the fourth feature information;

[0282] The sum of the absolute values ​​of the distance differences of the various operators is determined, and the sum of the absolute values ​​is used as the distance difference.

[0283] In a specific application scenario, the generating module 72 is specifically used for:

[0284] Obtaining superpixel matching scores between the plurality of third superpixels to be processed and the plurality of fourth superpixels to be processed;

[0285] a second adjustment parameter determined according to the superpixel matching score and the distance difference;

[0286] The matching score is adjusted using the second adjustment parameter.

[0287] In a specific application scenario, the generating module 72 is further used to:

[0288] respectively calculating adjustment factors of the first adjustment parameter and the second adjustment parameter;

[0289] Based on the adjustment factor, a third adjustment parameter is calculated by using the first adjustment parameter and the second adjustment parameter;

[0290] The generating module 72 is specifically configured to adjust the matching score using the third adjustment parameter.

[0291] In a specific application scenario, the acquisition module 71 is specifically used to:

[0292] A simple linear iterative clustering (SLIC) algorithm is used to perform superpixel segmentation on the pixels to be processed in the processed image to obtain superpixel information.

[0293] In a specific application scenario, the acquisition module 71 is specifically used to:

[0294] Performing color space conversion on the image to be processed to generate color space coordinates of the pixel to be processed;

[0295] In the color space coordinates, a simple linear iterative clustering algorithm is used to determine the color space coordinates of the superpixel center of the pixel to be processed.

[0296] Based on the above Figure 1 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 The method shown.

[0297] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0298] Based on the above Figure 1 The method shown, and Figure 7 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, such as a smart phone, a tablet computer, a drone, a smart robot and other smart terminals, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 The method shown.

[0299] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0300] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0301] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0302] Based on the above Figure 1 The method shown, and Figure 7 The virtual device embodiment shown in the embodiment further provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the above-mentioned Figure 1 The method shown.

[0303] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the solution of this embodiment, the pixels to be processed in the image to be processed can be firstly segmented into superpixels to obtain superpixel information, and the image to be processed includes a base frame image and a reference frame image, and then based on a semi-global matching algorithm, dense motion vector information is generated according to the superpixel information, and the dense motion vector information is used for the image fusion of the base frame image and the reference frame image. By applying the technical solution of the present application, after obtaining superpixel information from the image to be processed, and further using a two-dimensional semi-global matching algorithm, dense motion vector information as image fusion information is generated according to the superpixel information, which can effectively improve the acquisition speed of image fusion information, reduce computational complexity and operating power consumption, and thus effectively improve the efficiency of image fusion, and reduce the requirements for device hardware.

[0304] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0305] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A method for acquiring image information, characterized in that: include: Performing super-pixel segmentation on pixels to be processed in an image to be processed to obtain super-pixel information, wherein the image to be processed includes a benchmark frame image and a reference frame image; Based on a semi-global matching algorithm, dense motion vector information is generated according to the superpixel information, and the dense motion vector information is used for image fusion of the benchmark frame image and the reference frame image.

2. The method according to claim 1, characterized in that: The method of generating dense motion vector information based on the superpixel information based on the two-dimensional semi-global matching algorithm includes: Performing feature extraction on a first pixel to be processed in the image to be processed to obtain a first feature, wherein the first pixel to be processed is located in a first superpixel to be processed; Performing feature extraction on a second pixel to be processed in the image to be processed to obtain a second feature, wherein the second pixel to be processed is located in a second superpixel to be processed, and the second superpixel to be processed is an adjacent superpixel of the first superpixel to be processed; Calculating a matching score between the first feature and the second feature according to the superpixel information, wherein the matching score is used to represent a correlation between the first pixel to be processed and the second pixel to be processed; Cost aggregation is performed on the matching scores to generate dense motion vector information.

3. The method according to claim 2, characterized in that The calculating, according to the superpixel information, a matching score between the first feature and the second feature comprises: Confirming first superpixel information corresponding to the first feature; Confirming a plurality of second superpixel information corresponding to a plurality of the second features; Based on the Hamming distance, multiple matching scores between the first superpixel information and multiple second superpixel information are calculated.

4. The method according to claim 3, characterized in that: Before performing cost aggregation on the matching scores to generate dense motion vector information, the method further includes: Among the multiple matching scores, confirm the matching scores to be verified whose scores are lower than the preset scores and the matching result scores whose scores are higher than the preset scores; Extracting a target pixel in the first superpixel information to be verified corresponding to the matching score to be verified; Extracting the superpixel center of the second superpixel information to be verified corresponding to the matching score to be verified to obtain the superpixel center to be verified; Confirming the superpixel information matching result between the target pixel and the superpixel center to be verified; Based on the matching result, a first adjustment is made to the matching score between the first superpixel information to be verified and the second superpixel information to be verified; Using the first adjusted matching score as the matching result score; The cost aggregation of the matching scores to generate dense motion vector information includes: Cost aggregation is performed on the matching result scores to generate dense motion vector information.

5. The method according to claim 4, characterized in that The confirming the superpixel information matching result between the target pixel and the superpixel center to be verified includes: Calculate a first absolute value of a color distance difference between the target pixel and the center of the superpixel to be verified and a second absolute value of a spatial distance difference; Obtaining an absolute sum of the first absolute value and the second absolute value; The superpixel information matching result between the target pixel and the superpixel center to be verified is confirmed according to the absolute sum value.

6. The method according to claim 4, characterized in that The first adjusting the matching score between the first superpixel information to be verified and the second superpixel information to be verified based on the matching result includes: If the target pixel and the pixel to be verified correspond to each other, obtaining the absolute sum value; Adjust a first adjustment parameter between the first superpixel information and the second superpixel information according to the absolute sum value; According to the first adjustment parameter, a first adjustment is performed on the matching score between the first superpixel information to be verified and the second superpixel information to be verified.

7. The method according to claim 6, characterized in that The cost aggregation of the matching scores to generate dense motion vector information includes: Obtaining a plurality of third superpixels to be processed in the benchmark frame image and a plurality of fourth superpixels to be processed in the reference frame image, wherein the plurality of third superpixels to be processed and the plurality of fourth superpixels to be processed have the highest mutual matching scores; Determine a first edge intersection point between the plurality of third superpixels to be processed, and determine a second edge intersection point between the plurality of fourth superpixels to be processed; Calculating a distance difference between the first edge intersection point and the second edge intersection point; Performing a second adjustment on the matching score according to the distance difference; Cost aggregation is performed on the matching score adjusted for the second time to generate dense motion vector information.

8. The method according to claim 7, characterized in that The calculating the distance difference between the first edge intersection point and the second edge intersection point includes: Performing feature extraction on the first edge intersection to obtain third feature information; Performing feature extraction on the second edge intersection to obtain fourth feature information; Calculating the absolute value of the distance difference between the operators in the third feature information and the fourth feature information; The sum of the absolute values ​​of the distance differences of the various operators is determined, and the sum of the absolute values ​​is used as the distance difference.

9. The method according to claim 7, characterized in that: The adjusting the matching score according to the distance difference includes: Obtaining superpixel matching scores between the plurality of third superpixels to be processed and the plurality of fourth superpixels to be processed; a second adjustment parameter determined according to the superpixel matching score and the distance difference; The matching score is adjusted using the second adjustment parameter.

10. The method according to claim 9, characterized in that Before adjusting the matching score by using the second adjustment parameter, the method further includes: respectively calculating adjustment factors of the first adjustment parameter and the second adjustment parameter; Based on the adjustment factor, a third adjustment parameter is calculated by using the first adjustment parameter and the second adjustment parameter; The adjusting the matching score by using the second adjustment parameter includes: The matching score is adjusted using the third adjustment parameter.

11. The method according to claim 1, characterized in that: The step of performing super-pixel segmentation on the pixels to be processed in the image to be processed to obtain super-pixel information includes: A simple linear iterative clustering (SLIC) algorithm is used to perform superpixel segmentation on the pixels to be processed in the processed image to obtain superpixel information.

12. The method according to claim 11, characterized in that The method of using the SLIC algorithm to perform superpixel segmentation on the pixels to be processed in the image to be processed to obtain superpixel information includes: Performing color space conversion on the image to be processed to generate color space coordinates of the pixel to be processed; In the color space coordinates, a simple linear iterative clustering algorithm is used to determine the color space coordinates of the superpixel center of the pixel to be processed.

13. An image information acquisition device, characterized in that: include: An acquisition module, used for performing super-pixel segmentation on pixels to be processed in an image to be processed to obtain super-pixel information, wherein the image to be processed includes a benchmark frame image and a reference frame image; A generation module is used to generate dense motion vector information according to the superpixel information based on a semi-global matching algorithm, wherein the dense motion vector information is used for image fusion of the benchmark frame image and the reference frame image.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

16. A chip, characterized in that: The invention comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 12.

Images