A binocular image-based scene flow detection method, system, device and medium

Abstract

The application discloses a scene flow detection method and system based on binocular images, equipment and a medium, wherein the embodiment can improve the accuracy of feature extraction by extracting features of the binocular image set through a feature extraction model, and can improve the accuracy and stability of scene flow data by respectively performing disparity optimization processing and optical flow optimization processing on the extracted features to obtain scene flow data, so as to accurately obtain the position and motion information of a target, improve the real-time performance and accuracy of monitoring, and be widely applied in the technical field of computer vision.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device and medium for scene flow detection based on binocular images. Background Technology

[0002] With increasing traffic volume on highways, traffic conditions are becoming increasingly complex. To improve highway efficiency, more and more traffic incident detection technologies are being deployed. Compared to technologies using LiDAR and millimeter-wave radar for highway detection, camera-based detection algorithms can provide dense point clouds and visualized results, offering better support for high-level highway scheduling tasks. However, current camera-based highway detection primarily focuses on vehicle identification, struggling to obtain target location and motion information, resulting in poor real-time performance and accuracy in highway monitoring. Therefore, the technical problems existing in these technologies urgently need to be addressed. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a scene flow detection method, system, device and medium based on binocular images to obtain scene flow information.

[0004] On one hand, the present invention provides a scene flow detection method based on binocular images, including:

[0005] Obtain a set of stereo images;

[0006] The stereo image set is subjected to feature extraction using a feature extraction model to obtain an image feature set;

[0007] Disparity is calculated based on the image feature set, and disparity optimization is performed through cost aggregation to obtain disparity data;

[0008] Optical flow calculation is performed based on the image feature set, and optical flow optimization is performed using an energy function to obtain optical flow data;

[0009] The parallax data and the optical flow data are paired and processed to obtain scene flow data.

[0010] Optionally, acquiring the binocular image set includes:

[0011] A set of binocular images is acquired using a binocular camera or binocular video camera. The set of binocular images includes a current frame view and adjacent frame views. The current frame view includes a left view and a right view. The adjacent frame views are the previous frame view or the next frame view of the current frame view.

[0012] Optionally, the step of extracting features from the stereo image set using a feature extraction model to obtain an image feature set includes:

[0013] The binocular image set is subjected to image segmentation processing to obtain a segmented image set;

[0014] The segmented image set is input into the feature extraction model for convolution calculation, normalization calculation and binarization calculation to obtain the image feature set. The feature extraction model is constructed based on a binary convolutional network.

[0015] Optionally, the step of calculating disparity based on the image feature set and performing disparity optimization through cost aggregation to obtain disparity data includes:

[0016] Obtain an image feature set, which includes left view features and right view features;

[0017] The matching cost is calculated for the left view features and the right view features. The minimum cost is selected based on the calculated matching cost to obtain the initial disparity.

[0018] Based on the initial disparity, candidate disparities are selected, and cost aggregation is performed on the initial disparity and the candidate disparities. The minimum aggregated cost is selected as the disparity data.

[0019] Optionally, the step of performing optical flow calculation based on the image feature set and optimizing the optical flow using an energy function to obtain optical flow data includes:

[0020] Obtain an image feature set, the image feature set including a first feature and a second feature;

[0021] The first feature and the second feature are subjected to feature block matching processing to obtain a matching feature block;

[0022] Optical flow is calculated based on the matching feature blocks to obtain the initial optical flow;

[0023] An energy function is constructed based on the initial optical flow, the first feature, and the second feature;

[0024] The initial optical flow is restored and matched according to the energy function to obtain optical flow data.

[0025] Optionally, the step of performing image segmentation processing on the stereo image set to obtain a segmented image set includes:

[0026] Each image in the binocular image set is processed to select pixels, resulting in a pixel set.

[0027] The image is divided into segments centered on each pixel in the pixel set.

[0028] Optionally, the step of selecting candidate disparities based on the initial disparity, performing cost aggregation processing on the initial disparity and the candidate disparities, and selecting the minimum aggregated cost as disparity data includes:

[0029] The initial disparity is subjected to disparity optimization processing, and the optimized disparity is selected to obtain candidate disparities;

[0030] Cost aggregation is performed on the initial disparity and the candidate disparities, and the minimum aggregation cost is selected as the disparity data.

[0031] On the other hand, embodiments of the present invention also provide a scene flow detection system based on binocular images, comprising:

[0032] The first module is used to acquire a set of stereo images;

[0033] The second module is used to extract features from the stereo image set using a feature extraction model to obtain an image feature set.

[0034] The third module is used to perform disparity calculation based on the image feature set and to perform disparity optimization processing through cost aggregation to obtain disparity data.

[0035] The fourth module is used to perform optical flow calculation based on the image feature set and to perform optical flow optimization processing through an energy function to obtain optical flow data;

[0036] The fifth module is used to pair the disparity data and the optical flow data to obtain scene flow data.

[0037] Optionally, the first module is used to acquire a set of stereo images, including:

[0038] The first unit is used to acquire a set of binocular images through a binocular camera or a binocular webcam. The set of binocular images includes a current frame view and adjacent frame views. The current frame view includes a left view and a right view. The adjacent frame views are the previous frame view or the next frame view of the current frame view.

[0039] Optionally, the second module is used to extract features from the stereo image set using a feature extraction model to obtain an image feature set, including:

[0040] The second unit is used to perform image segmentation processing on the binocular image set to obtain a segmented image set;

[0041] The third unit is used to input the segmented image set into the feature extraction model for convolution calculation, normalization calculation and binarization calculation to obtain the image feature set. The feature extraction model is constructed based on a binary convolutional network.

[0042] Optionally, the third module is used to perform disparity calculation based on the image feature set, and to perform disparity optimization processing through cost aggregation to obtain disparity data, including:

[0043] The fourth unit is used to obtain an image feature set, which includes left view features and right view features;

[0044] The fifth unit is used to calculate the matching cost of the left view features and the right view features, and select the minimum cost based on the calculated matching cost to obtain the initial disparity.

[0045] The sixth unit is used to select candidate disparities based on the initial disparity, perform cost aggregation processing on the initial disparity and the candidate disparities, and select the minimum aggregation cost as disparity data.

[0046] Optionally, the fourth module is used to perform optical flow calculation based on the image feature set and to perform optical flow optimization processing through an energy function to obtain optical flow data, including:

[0047] The seventh unit is used to acquire an image feature set, which includes a first feature and a second feature;

[0048] The eighth unit is used to perform feature block matching processing on the first feature and the second feature to obtain a matching feature block;

[0049] The ninth unit is used to perform optical flow calculation based on the matching feature blocks to obtain the initial optical flow;

[0050] The tenth unit is used to construct an energy function based on the initial optical flow, the first feature, and the second feature;

[0051] The eleventh unit is used to restore and match the initial optical flow according to the energy function to obtain optical flow data.

[0052] Optionally, the first unit is configured to perform image segmentation processing on the binocular image set to obtain a segmented image set, including:

[0053] Each image in the binocular image set is processed to select pixels, resulting in a pixel set.

[0054] The image is divided into segments centered on each pixel in the pixel set.

[0055] Optionally, the sixth unit is configured to select candidate disparities based on the initial disparity, perform cost aggregation processing on the initial disparity and the candidate disparities, and select the minimum aggregated cost as disparity data, including:

[0056] The initial disparity is subjected to disparity optimization processing, and the optimized disparity is selected to obtain candidate disparities;

[0057] Cost aggregation is performed on the initial disparity and the candidate disparities, and the minimum aggregation cost is selected as the disparity data.

[0058] On the other hand, embodiments of the present invention also disclose an electronic device, including a processor and a memory;

[0059] The memory is used to store programs;

[0060] The processor executes the program to implement the method described above.

[0061] On the other hand, embodiments of the present invention also disclose a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0062] On the other hand, embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0063] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: The embodiments of the present invention perform feature extraction on the binocular image set through a feature extraction model, which can improve the accuracy of feature extraction. The extracted features are then subjected to disparity optimization processing and optical flow optimization processing to obtain scene flow data, which can improve the accuracy and stability of scene flow data, thereby accurately obtaining the position and motion information of the target and improving the real-time performance and accuracy of monitoring. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1This is a flowchart of a scene flow detection method based on binocular images provided in an embodiment of this application;

[0066] Figure 2 This is a schematic diagram of the structure of a scene flow detection system based on binocular images provided in an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the structure of a scene flow detection device based on binocular images provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] In related technologies, comprehensive highway information detection generally employs highway detection algorithms based on sparse point clouds, such as those using LiDAR or millimeter-wave radar, or video-based detection algorithms. Compared to these algorithms, video-based algorithms can provide dense point clouds and visualized results, offering better support for high-level highway scheduling tasks. However, current video-based highway detection primarily focuses on vehicle identification, struggling to obtain target location and movement information, resulting in poor real-time performance and accuracy in highway monitoring.

[0070] Reference Figure 1 This invention provides a scene flow detection method based on binocular images, comprising:

[0071] S101. Obtain the set of stereo images;

[0072] S102. Extract features from the stereo image set using a feature extraction model to obtain an image feature set;

[0073] S103. Perform disparity calculation based on the image feature set, and perform disparity optimization processing through cost aggregation to obtain disparity data;

[0074] S104. Perform optical flow calculation based on the image feature set, and perform optical flow optimization processing through energy function to obtain optical flow data;

[0075] S105. The parallax data and the optical flow data are paired to obtain scene flow data.

[0076] In this embodiment of the invention, a binocular image set is acquired through a binocular camera or binocular camera module, resulting in a real-time view image stream, including a left view and a right view. The left and right views are then input into a feature extraction model for feature extraction. This feature extraction model is constructed using a binary convolutional network, which improves the accuracy of feature extraction. Disparity is then calculated on the left and right view features in the image feature set output by the feature extraction model, and disparity optimization is performed through cost aggregation to obtain optimized disparity data. Simultaneously, optical flow is calculated on the left view features and the previous frame's left view features in the image feature set output by the feature extraction model, and optical flow optimization is performed through an energy function to obtain optimized optical flow data. Finally, the optimized disparity data and optimized optical flow data are paired and packaged to obtain scene stream data.

[0077] As a further preferred embodiment, in step S101 above, acquiring the binocular image set includes:

[0078] A set of binocular images is acquired using a binocular camera or binocular video camera. The set of binocular images includes a current frame view and adjacent frame views. The current frame view includes a left view and a right view. The adjacent frame views are the previous frame view or the next frame view of the current frame view.

[0079] In this embodiment of the invention, a binocular image of the current frame can be acquired using a binocular camera or binocular video camera. To facilitate subsequent optical flow calculations, adjacent frame views are used for calculation, and these adjacent frame views are saved and acquired. That is, the binocular image set includes the current frame view (left and right views) and adjacent frame views (the frame before or after the left view and the frame before or after the right view). It is conceivable that this embodiment can also acquire a video stream using a binocular camera or binocular video camera, and extract the binocular image set from each frame of the video stream.

[0080] Further, as a preferred embodiment, the step of extracting features from the stereo image set using a feature extraction model to obtain an image feature set includes:

[0081] The binocular image set is subjected to image segmentation processing to obtain a segmented image set;

[0082] The segmented image set is input into the feature extraction model for convolution calculation, normalization calculation and binarization calculation to obtain the image feature set. The feature extraction model is constructed based on a binary convolutional network.

[0083] In this embodiment of the invention, each frame of the stereo image set is segmented, specifically by a preset size, to obtain a segmented image set. This segmented image set includes segmented image blocks for each frame of the stereo image set. The segmented image set is then input into a feature extraction model, which is constructed based on a binary convolutional network. This feature extraction module includes two identical binary neural networks. Its input consists of two 7×7 image blocks, representing the left and right view images respectively, and its output is the corresponding extracted 64-bit binary feature vector. The feature extraction module consists of three convolutional layers. Each layer's calculation does not require pixel padding and includes three main steps: convolution calculation, normalization calculation, and binarization calculation. The first layer, in addition to the above three steps, also includes a grayscale binarization and restoration operation, with a convolution output feature size of 5×5×64. The second layer's convolution output feature size is 3×3×64, and the last layer outputs a 64-bit binary feature vector. The first convolutional layer takes an 8-bit grayscale image block as input. In the calculation, the corresponding bits and weights are directly convolved. The convolution kernel size is 3×3. Since different bits of the input 8-bit grayscale value represent different values, direct bitwise convolution will lose this bit information. To solve this problem, after the convolution operation, a restoration operation is performed on the binary features of different bit depths, specifically expressed by the following formula:

[0084]

[0085] Where S represents the calculation result after convolution and deconvolution, n represents the number of bits of the corresponding pixel, i represents the pixel in the image block, x represents the gray value of the pixel, w represents the corresponding convolution weight, and b represents...

[0086] After the convolution calculation is completed, the result is normalized and binarized before being input into the second layer. The inputs to the second and third layers are binary features, so the restoration operation in the first layer can be avoided during the calculation, and the convolution operation can be performed directly. Finally, the third layer periodically outputs two 64-bit binary features as the output of the entire feature extraction model.

[0087] Further, as a preferred embodiment, the step of calculating disparity based on the image feature set and performing disparity optimization processing through cost aggregation to obtain disparity data includes:

[0088] Obtain an image feature set, which includes left view features and right view features;

[0089] The matching cost is calculated for the left view features and the right view features. The minimum cost is selected based on the calculated matching cost to obtain the initial disparity.

[0090] Based on the initial disparity, candidate disparities are selected, and cost aggregation is performed on the initial disparity and the candidate disparities. The minimum aggregated cost is selected as the disparity data.

[0091] In this embodiment of the invention, an image feature set is first obtained, which includes left view features and right view features, i.e., left view features and right view features obtained by feature extraction of the left and right views of the current frame through a feature extraction model. Then, matching cost calculation is performed on the left view features and right view features, and disparity search is performed within a small search range. In this embodiment of the invention, a maximum search range D is set. max =64, the specific operation is as follows: within the maximum search range, calculate the matching cost. The matching cost is calculated as follows:

[0092] C=∑f l ×f r ;

[0093] Where C is the calculated cost, and fl and fr are the left and right view features output by the feature extraction module.

[0094] The matching cost is calculated using the above formula, and the disparity with the minimum cost is selected as the initial disparity. The first disparity optimization is then performed based on the initial disparity. Next, for a pixel (x, y) selected in either the left or right view, five alternative disparities s(d) (d1, d2, d3, d4, d5) are chosen for cost calculation when calculating the disparity at position (x, y). The chosen alternative disparities are: d1 is the initial disparity at position (x, y); d2 is the optimized disparity at position (x-1, y) (i.e., the pixel to the left of the current disparity); d3 is the optimized disparity at position (x, y-1) (i.e., the pixel directly above the current disparity); d4 is the initial disparity at position (x-1, y) (i.e., the pixel to the left of the current disparity); and d5 is the initial disparity at position (x-1, y) (i.e., the second pixel to the left of the current disparity).

[0095] The matching cost is calculated based on the five alternative disparities obtained in the previous step, denoted as the cost space C. s(d) The specific calculation formula is as follows:

[0096] C s(d) =∑f l (x,y)×f r (xs(d));

[0097] Cost aggregation is performed based on the cost calculated using the above formula. Specifically, a rectangular cost space with side length Q is selected in the cost space, and the costs within the rectangle with side length Q are summed to obtain the aggregated cost CA. s(d) The value of Q determines the aggregation effect and can be selected according to specific circumstances. The specific calculation formula is as follows:

[0098] CA s(d) =∑ Q C s(d) ;

[0099] The disparity with the lowest cost after aggregation is selected as the optimized disparity for a second disparity optimization. The second disparity optimization step involves repeating the first disparity optimization algorithm and using the result of the second disparity optimization as the disparity output for binocular matching.

[0100] As a further preferred embodiment, the step of performing optical flow calculation based on the image feature set and optimizing the optical flow through an energy function to obtain optical flow data includes:

[0101] Obtain an image feature set, the image feature set including a first feature and a second feature;

[0102] The first feature and the second feature are subjected to feature block matching processing to obtain a matching feature block;

[0103] Optical flow is calculated based on the matching feature blocks to obtain the initial optical flow;

[0104] An energy function is constructed based on the initial optical flow, the first feature, and the second feature;

[0105] The initial optical flow is restored and matched according to the energy function to obtain optical flow data.

[0106] In this embodiment of the invention, an image feature set is obtained, which includes a first feature and a second feature. The first feature can be a left view feature or a right view feature, and the second feature is the corresponding previous frame view feature or the next frame image feature. In this embodiment, the first feature is a left view feature, and the second feature is the previous frame left view feature. Since optical flow estimation matches the previous and next frames, the features of the left image need to be stored after output. This embodiment also improves a storage module based on Direct Memory Access (DMA). Specifically, every four 64-bit features are concatenated into a 256-bit character and stored in the corresponding memory. In storage, 16 features are allocated at a time, so the address bits increase by 16 at a time. Therefore, each DMA storage operation stores 4096 bits of data, for a total of 64 features. To avoid affecting the matching, the feature storage module allocates 10 separate memory spaces for feature storage. These 10 address spaces are completely independent and do not overlap. Optical flow estimation requires obtaining the features of the current binocular left image, i.e., the first feature is denoted as f. l t And the features of the left image of the previous frame stored in memory, the second feature is denoted as f. l t-1 .

[0107] Next, f l t and f l t-1 After clock alignment, first adjust f l t and f l t-1 Feature block matching is performed. Specifically, four features corresponding to pixels (x, y), (x+1, y), (x, y+1), and (x+1, y+1) are selected as a feature block P1. Within a matching range of k×k, to reduce computational overhead, this invention chooses a k value of 7. All feature blocks P1 are obtained by sliding a window with a step size of 2 and a window size of 2×2. n Cost of performing matching calculations:

[0108] C x,y =∑P1×P n ;

[0109] The position corresponding to the feature block with the lowest cost is selected, and the position (x,y) of the current pixel is subtracted to obtain a two-dimensional vector V, which is used as the initial optical flow.

[0110] Finally, using vectors V and f l t and f l t-1 A new energy function E is constructed, specifically by averaging nine vectors within a 3×3 region centered at position (x,y) based on the assumption that similar optical flows are invariant in small time and small displacements, and then using V1 as a candidate vector in the feature map f. l t and f l t-1 The process involves restoring and matching the vector. The restoration method multiplies the vector by the step size. After restoration, the matching process is repeated according to the above steps to obtain a refined optical flow as the output, resulting in optimized optical flow data.

[0111] As can be imagined, this embodiment outputs a total of 24 bits of data through optical flow optimization and parallax optimization. The first 8 bits are parallax data, which is an unsigned integer. The middle 8 bits and the last 8 bits store the x and y components of the optical flow, respectively, which are two signed integers. The parallax information and the optical flow information are paired and packaged to output a triple as the output scene flow information.

[0112] Reference Figure 2 This invention also provides a scene flow detection system based on binocular images, comprising:

[0113] The first module 201 is used to acquire a set of stereo images;

[0114] The second module 202 is used to extract features from the stereo image set using a feature extraction model to obtain an image feature set.

[0115] The third module 203 is used to perform disparity calculation based on the image feature set and to perform disparity optimization processing through cost aggregation to obtain disparity data.

[0116] The fourth module 204 is used to perform optical flow calculation based on the image feature set and to perform optical flow optimization processing through an energy function to obtain optical flow data;

[0117] The fifth module 205 is used to pair the disparity data and the optical flow data to obtain scene flow data.

[0118] Reference Figure 3 This invention also provides an electronic device, including a processor 301 and a memory 302; the memory is used to store a program; the processor executes the program to implement the method described above.

[0119] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0120] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0121] In summary, the embodiments of the present invention have the following advantages: the embodiments of the present invention improve the accuracy and stability of matching and optical flow estimation by acquiring binocular images, extracting features from the left and right images and the previous and next frames of the binocular images, performing disparity optimization and optical flow optimization processing respectively, and improving the accuracy and stability of matching and optical flow estimation by using features extracted based on convolutional neural networks.

[0122] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0123] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0124] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0126] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0127] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0128] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0129] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0130] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A scene flow detection method based on binocular images, characterized in that, The method includes: Obtain a set of stereo images; The stereo image set is subjected to feature extraction using a feature extraction model to obtain an image feature set; Disparity is calculated based on the image feature set, and disparity optimization is performed through cost aggregation to obtain disparity data; Optical flow calculation is performed based on the image feature set, and optical flow optimization is performed using an energy function to obtain optical flow data; The parallax data and the optical flow data are paired to obtain scene flow data; The step of extracting features from the stereo image set using a feature extraction model to obtain an image feature set includes: The binocular image set is subjected to image segmentation processing to obtain a segmented image set; The segmented image set is input into the feature extraction model for convolution, normalization, and binarization calculations to obtain an image feature set. The feature extraction model is constructed based on a binary convolutional network. The feature extraction model consists of three convolutional layers. The first convolutional layer is used for convolution, normalization, binarization, and grayscale binarization and restoration operations. The expression for the restoration operation is as follows: Where S represents the calculation result after convolution and deconvolution, n represents the number of bits of the corresponding pixel, i represents the corresponding pixel in the image patch, x represents the gray value of the pixel, and w represents the corresponding convolution weight. The step of calculating optical flow based on the image feature set and optimizing optical flow using an energy function to obtain optical flow data includes: Obtain an image feature set, the image feature set including a first feature and a second feature; The first feature and the second feature are subjected to feature block matching processing to obtain a matching feature block; The initial optical flow is obtained by calculating the optical flow based on the matched feature blocks; An energy function is constructed based on the initial optical flow, the first feature, and the second feature; The initial optical flow is restored and matched according to the energy function to obtain optical flow data.

2. The method according to claim 1, characterized in that, The acquisition of the binocular image set includes: A set of binocular images is acquired using a binocular camera or binocular video camera. The set of binocular images includes a current frame view and adjacent frame views. The current frame view includes a left view and a right view. The adjacent frame views are the previous frame view or the next frame view of the current frame view.

3. The method according to claim 1, characterized in that, The step of calculating disparity based on the image feature set and optimizing disparity through cost aggregation to obtain disparity data includes: Obtain an image feature set, which includes left view features and right view features; The matching cost is calculated for the left view features and the right view features. The minimum cost is selected based on the calculated matching cost to obtain the initial disparity. Based on the initial disparity, candidate disparities are selected, and cost aggregation is performed on the initial disparity and the candidate disparities. The minimum aggregated cost is selected as the disparity data.

4. The method according to claim 1, characterized in that, The step of performing image segmentation processing on the stereo image set to obtain a segmented image set includes: Each image in the binocular image set is processed to select pixels, resulting in a pixel set. The image is divided into segments centered on each pixel in the pixel set.

5. The method according to claim 3, characterized in that, The step of selecting candidate disparities based on the initial disparity, performing cost aggregation on the initial disparity and the candidate disparities, and selecting the minimum aggregation cost as the disparity data includes: The initial disparity is subjected to disparity optimization processing, and the optimized disparity is selected to obtain candidate disparities; Cost aggregation is performed on the initial disparity and the candidate disparities, and the minimum aggregation cost is selected as the disparity data.

6. A scene flow detection system based on binocular images, characterized in that, The system includes: The first module is used to acquire a set of stereo images; The second module is used to extract features from the stereo image set using a feature extraction model to obtain an image feature set. The third module is used to perform disparity calculation based on the image feature set and to perform disparity optimization processing through cost aggregation to obtain disparity data. The fourth module is used to perform optical flow calculation based on the image feature set and to perform optical flow optimization processing through an energy function to obtain optical flow data; The fifth module is used to pair the disparity data and the optical flow data to obtain scene flow data; The second module is used to extract features from the stereo image set using a feature extraction model to obtain an image feature set, including: The binocular image set is subjected to image segmentation processing to obtain a segmented image set; The segmented image set is input into the feature extraction model for convolution, normalization, and binarization calculations to obtain an image feature set. The feature extraction model is constructed based on a binary convolutional network. The feature extraction model consists of three convolutional layers. The first convolutional layer is used for convolution, normalization, binarization, and grayscale binarization and restoration operations. The expression for the restoration operation is as follows: Where S represents the calculation result after convolution and deconvolution, n represents the number of bits of the corresponding pixel, i represents the corresponding pixel in the image patch, x represents the gray value of the pixel, and w represents the corresponding convolution weight. The step of calculating optical flow based on the image feature set and optimizing optical flow using an energy function to obtain optical flow data includes: Obtain an image feature set, the image feature set including a first feature and a second feature; The first feature and the second feature are subjected to feature block matching processing to obtain a matching feature block; The initial optical flow is obtained by calculating the optical flow based on the matched feature blocks; An energy function is constructed based on the initial optical flow, the first feature, and the second feature; The initial optical flow is restored and matched according to the energy function to obtain optical flow data.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store programs; The processor executes the program to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images