Parallax processing method and parallax processing system

Abstract

The application discloses a disparity processing method and a disparity processing system. It relates to the field of automatic driving, and comprises the following steps: processing binocular images through a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than that of the binocular images; performing disparity optimization on the initial disparity image through a first convolution layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than that of the initial disparity image; and calculating the first disparity image and a weight feature image to obtain a target disparity image, wherein the weight feature image is obtained by processing a target image in the binocular images through the reference disparity network, and the resolution of the target disparity image is higher than that of the first disparity image. Through the application, the technical problem of poor accuracy of disparity values in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and more specifically, to a parallax processing method and a parallax processing system. Background Technology

[0002] In the field of assisted driving technology, stereo vision is a key technology that aims to recover the three-dimensional depth information of a scene by calculating pixel-level parallax using the imaging differences of binocular cameras, mimicking the working principle of biological eyes. This technology plays an indispensable role in many cutting-edge applications such as robot navigation, obstacle avoidance, and SLAM (Simultaneous Localization and Mapping).

[0003] However, despite the increasing maturity of stereo vision theory, practical deployment, especially on resource-constrained embedded platforms, still faces numerous challenges. Related technologies utilize hardware resources with high computing power and heterogeneous acceleration capabilities; however, some hardware resources, due to their lack of support for specific operators int16 quantization, experience a significant decrease in accuracy when processing disparity maps. When disparity values ​​are forced to decrease from the int16 quantization level to int8, the representation range of disparity information shrinks drastically, directly affecting the output accuracy of stereo matching algorithms, and potentially reducing the reliability and safety of driver assistance systems.

[0004] There is currently no effective solution to the technical problem of poor accuracy of parallax values ​​in related technologies. Summary of the Invention

[0005] The main objective of this application is to provide a parallax processing method and a parallax processing system to solve the technical problem of poor accuracy of parallax values ​​in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a disparity processing method is provided. The method includes: processing a binocular image using a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image; performing disparity optimization on the initial disparity image using a first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; and calculating a target disparity image by combining the first disparity image and a weighted feature image, wherein the weighted feature image is obtained by processing a target image in the binocular image using the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image.

[0007] Furthermore, after performing disparity optimization on the initial disparity image through the first convolutional layer to obtain the first disparity image, the method further includes: transmitting the first disparity image and the weighted feature image to the central processing unit of the target device, wherein the baseline disparity network and the first convolutional layer are deployed in the neural network acceleration processor of the target device.

[0008] Further, calculating the first disparity image and the weighted feature image to obtain the target disparity image includes: in the central processing unit, converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively; copying each data point in the first floating-point data to a vector of a target number of bits using a first vector instruction value to obtain processed first floating-point data; and calculating the processed first floating-point data and the second floating-point data using a second vector instruction value to obtain the target disparity image.

[0009] Further, calculating the target disparity image by using the second vector instruction value on the processed first floating-point data and the second floating-point data includes: performing element-wise multiplication on the processed first floating-point data and the second floating-point data to obtain a first calculation result; adding the target dimension data in the first calculation result to obtain a second calculation result; and inverse quantizing the second calculation result to obtain the target disparity image.

[0010] Further, converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively includes: obtaining a first quantization coefficient of the first disparity image and a second quantization coefficient corresponding to the weighted feature image, wherein the first quantization coefficient is obtained when processing a binocular image of a first resolution through a reference disparity network, and the second quantization coefficient is obtained when processing a target image of the binocular image through the reference disparity network; calculating the first floating-point data by performing calculations on the first disparity image and the first quantization coefficient; and calculating the second floating-point data by performing calculations on the weighted feature image and the second quantization coefficient.

[0011] Furthermore, the method further includes: extracting features from the target image through the backbone network in the benchmark disparity network to obtain a first feature vector; fusing features from the first feature vector through the feature pyramid network in the benchmark disparity network to obtain a second feature vector; performing convolution calculation and upsampling on the second feature vector to obtain a third feature vector; and obtaining the weighted feature image based on the third feature vector.

[0012] Further, calculating the target disparity image by processing the first disparity image and the weighted feature image includes: processing the first disparity image to obtain a second disparity image, wherein the resolution of the second disparity image is higher than the resolution of the first disparity image; and calculating the target disparity image by processing the second disparity image and the weighted feature image.

[0013] Further, processing the first disparity image to obtain the second disparity image includes: calculating the first disparity image through a second convolutional layer to obtain a third calculation result, wherein the resolution of the third calculation result is the same as the resolution of the first disparity image; reshaping and sorting the third calculation result to obtain a fourth calculation result, wherein the resolution of the fourth calculation result is higher than the resolution of the third calculation result; calculating the fourth calculation result through a third convolutional layer to obtain a fifth calculation result, wherein the resolution of the fifth calculation result is the same as the resolution of the fourth calculation result; and reshaping and sorting the fifth calculation result to obtain the second disparity image.

[0014] To achieve the above objectives, according to another aspect of this application, a disparity processing system is provided, comprising: a neural network acceleration processor, wherein the neural network acceleration processor is deployed with a reference disparity network and a first convolutional layer, wherein the reference disparity network processes a binocular image to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image, and the first convolutional layer performs disparity optimization on the initial disparity image to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; a central processing unit, wherein the central processing unit calculates a target disparity image by processing the first disparity image and a weighted feature image, wherein the weighted feature image is obtained by processing the target image of the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image.

[0015] To achieve the above objectives, according to another aspect of this application, a disparity processing apparatus is provided. The apparatus includes: a processing unit for processing a binocular image through a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image; an optimization unit for performing disparity optimization on the initial disparity image through a first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; and a calculation unit for calculating a target disparity image by processing the first disparity image and a weighted feature image, wherein the weighted feature image is obtained by processing a target image in the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image.

[0016] Furthermore, the device further includes: a transmission unit, configured to perform disparity optimization on the initial disparity image through a first convolutional layer to obtain a first disparity image, and then transmit the first disparity image and the weight feature image to the central processing unit of the target device, wherein the reference disparity network and the first convolutional layer are deployed in the neural network acceleration processor of the target device.

[0017] Further, the calculation unit includes: a conversion subunit, configured to convert the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively in the central processing unit; a copying subunit, configured to copy each data point in the first floating-point data to a vector of a target number of bits using a first vector instruction value to obtain processed first floating-point data; and a first calculation subunit, configured to perform calculations on the processed first floating-point data and the second floating-point data using a second vector instruction value to obtain the target disparity image.

[0018] Further, the first calculation subunit includes: a first processing module, used to perform element-wise multiplication of the processed first floating-point data and the second floating-point data to obtain a first calculation result; a second processing module, used to perform addition of the target dimension data in the first calculation result to obtain a second calculation result; and a quantization module, used to perform inverse quantization on the second calculation result to obtain the target disparity image.

[0019] Further, the conversion subunit includes: an acquisition module, configured to acquire a first quantization coefficient of the first disparity image and a second quantization coefficient corresponding to the weight feature image, wherein the first quantization coefficient is obtained when processing a binocular image of a first resolution through a reference disparity network, and the second quantization coefficient is obtained when processing a target image of the binocular image through the reference disparity network; a first calculation module, configured to calculate the first disparity image and the first quantization coefficient to obtain the first floating-point data; and a second calculation module, configured to calculate the weight feature image and the second quantization coefficient to obtain the second floating-point data.

[0020] Furthermore, the apparatus further includes: an extraction subunit, configured to extract features from the target image through the backbone network in the reference disparity network to obtain a first feature vector; a fusion subunit, configured to fuse the first feature vector through the feature pyramid network in the reference disparity network to obtain a second feature vector; a first processing subunit, configured to perform convolution calculation and upsampling processing on the second feature vector to obtain a third feature vector; and a second calculation subunit, configured to obtain the weighted feature image based on the third feature vector.

[0021] Furthermore, the calculation unit includes: a second processing subunit, used to process the first disparity image to obtain a second disparity image, wherein the resolution of the second disparity image is higher than the resolution of the first disparity image; and a third calculation subunit, used to calculate the second disparity image and the weighted feature image to obtain a target disparity image.

[0022] Further, the second processing subunit includes: a third calculation module, used to calculate the first disparity image through a second convolutional layer to obtain a third calculation result, wherein the resolution of the third calculation result is the same as the resolution of the first disparity image; a third processing module, used to reshape and sort the third calculation result to obtain a fourth calculation result, wherein the resolution of the fourth calculation result is higher than the resolution of the third calculation result; a fourth calculation module, used to calculate the fourth calculation result through a third convolutional layer to obtain a fifth calculation result, wherein the resolution of the fifth calculation result is the same as the resolution of the fourth calculation result; and a fourth processing module, used to reshape and sort the fifth calculation result to obtain the second disparity image.

[0023] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the parallax processing method of any one of the above-mentioned methods when it runs.

[0024] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a program, wherein the program controls the device where the storage medium is located to execute the parallax processing method described above when it is running.

[0025] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, wherein the computer program or instructions, when executed by a processor, are the parallax processing method described in any of the above.

[0026] In this embodiment, the following steps are employed: First, a reference disparity network is used to process the binocular image to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image; second, a first disparity image is obtained by performing disparity optimization on the initial disparity image through a first convolutional layer, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; third, a target disparity image is obtained by calculating the first disparity image and a weighted feature image, wherein the weighted feature image is obtained by processing the target image in the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image. This solves the technical problem of poor accuracy of disparity values ​​in related technologies.

[0027] In this application, a baseline disparity network processes the binocular images (left and right images) to obtain a low-resolution initial disparity image. A first convolutional layer is then used to optimize the initial disparity image, improving its resolution and generating a higher-resolution first disparity image. The low-resolution disparity image is upsampled and refined to compensate for the accuracy loss caused by previous quantization and to attempt to recover more details. The first disparity image is then fused with a weighted feature image to obtain the target disparity image, further improving the disparity resolution. The weighted feature image, derived from feature extraction of the target image by the baseline disparity network, carries additional information about image structure and texture. By multiplying it with the first disparity image, the weighted feature image helps to correct and refine the disparity estimate, thereby generating a disparity map with higher accuracy and resolution. The fusion process ensures that the disparity calculation utilizes information from the high-resolution image while also considering the coarse estimate from the low-resolution disparity map, balancing computational efficiency and output quality. This application specifically overcomes the parallax accuracy problem caused by quantization and dequantization in the prior art. Through multi-stage resolution enhancement and feature fusion, it improves the accuracy of the final parallax image and provides higher quality 3D information for downstream tasks. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0029] Figure 1 A hardware structure block diagram of a computer terminal for implementing a parallax processing method is shown.

[0030] Figure 2 This is a flowchart of a parallax processing method provided according to an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of the parallax processing method provided in the embodiments of this application. Figure 1 ;

[0032] Figure 4 This is a schematic diagram of the parallax processing method provided in the embodiments of this application. Figure 2 ;

[0033] Figure 5 This is a schematic diagram of the parallax processing method provided in the embodiments of this application. Figure 3 ;

[0034] Figure 6 This is a schematic diagram of the parallax processing method provided in the embodiments of this application. Figure 4 ;

[0035] Figure 7 This is a schematic diagram of a parallax processing system provided according to an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of a parallax processing device provided according to an embodiment of this application;

[0037] Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0041] Example 1

[0042] According to an embodiment of this application, a parallax processing method embodiment is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a parallax processing method is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the parallax processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the parallax processing method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0047] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The parallax processing method shown. Figure 2 This is a flowchart of a parallax processing method according to Embodiment 1 of this application. The parallax processing method includes:

[0049] Step S201: The binocular image is processed by a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than that of the binocular image.

[0050] Optionally, in the field of assisted driving, hardware chip resources with high computing power and heterogeneous acceleration capabilities are generally used for stereo matching. For example, the aforementioned reference disparity network can be deployed in a chip with high computing power and heterogeneous acceleration capabilities. However, this chip does not support int16 quantization of the upsampling operator. Therefore, the resolution of the disparity obtained by stereo matching on this chip will be relatively low. For example, the disparity value is automatically quantized to int8, and the range of int8 is much smaller than the true range of the disparity value, which leads to a sharp drop in disparity accuracy.

[0051] For example, a pair of stereo images from different viewpoints of the same scene are input into a baseline disparity network to calculate the left and right disparity for each pixel. The baseline disparity network first extracts features from the stereo images and can perform feature fusion through mechanisms such as skip connections or Feature Pyramid Networks (FPNs) to capture multi-layered information about the scene. By comparing the fused features of the stereo images, a cost volume is constructed, which contains the matching cost at different disparity values. Based on the cost volume, a low-resolution initial disparity image is obtained.

[0052] For example, the resolution of the input binocular image is 1. 3 H W, the left and right images are used to extract features using a baseline disparity network, and then feature fusion is performed to obtain a resolution lower than 1. 3 H W's 1 64 H / 4 The W / 4 feature is used to calculate the correlation between the extracted left and right image features, resulting in the Cost volume. Then, cost aggregation is performed on the Cost volume (a measure of cost or difference between corresponding pixels in two stereo images, typically the left and right images) to obtain 1. 36 H / 4 W / 4 (1 36 H / 4 W / 4 and 1 64 H / 4 Features with the same resolution (W / 4) are then used to perform softmax on the first dimension of cost aggregation and summed to obtain 1. 1 H / 4 W / 4 (1 1 H / 4 W / 4 and 1 36 H / 4 The initial disparity Disp_low (i.e., the low-resolution initial disparity image mentioned above) is calculated with the same resolution as W / 4, where the image width is W and the height is H.

[0053] Step S202: Perform disparity optimization on the initial disparity image through the first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image.

[0054] Optionally, the first convolutional layer upsamples the low-resolution initial disparity image to a higher resolution to obtain the first disparity image. For example, Disp_low (i.e., the aforementioned low-resolution initial disparity image) is upsampled using a fixed weight of 3. Unfolding the 3rd convolution (i.e., the first convolutional layer) yields Disp_low_unfold, with a resolution of 1. 9 H / 4 W / 4 (lower than the original resolution of the binocular image).

[0055] Step S203: Calculate the first disparity image and the weighted feature image to obtain the target disparity image. The weighted feature image is obtained by processing the target image in the binocular image through the reference disparity network. The resolution of the target disparity image is higher than that of the first disparity image.

[0056] Optionally, the target image in the binocular images is processed through a reference parallax network, for example, for the left image 1. 36 H / 4 The W / 4 features are convolved twice and upsampled to increase their resolution to the original image size, resulting in feature weights with a resolution of 1. 9 H W is obtained by calculating the weighted feature image by applying softmax to the weighted features. For example, applying softmax to the weighted features yields weight_unfold (i.e., the aforementioned weighted feature image), with a resolution of 1. 9 H The main purpose of the W. softmax operation is to convert it into a probability distribution that reflects the relative importance or weight of each pixel in the image when calculating the final disparity value. The softmax function can be applied to the second dimension and then the nine feature values ​​of the second dimension are converted into a probability distribution, ensuring that all values ​​add up to 1 and each value is between (0) and (1).

[0057] After obtaining the aforementioned weighted feature image, the final target disparity image is generated by multiplying the first disparity image and the weighted feature image element by element.

[0058] In an optional embodiment, if the resolution of the first disparity image is lower than the resolution corresponding to the weight feature image, the resolution of the first disparity image can be increased to the resolution corresponding to the weight feature image by a convolutional layer, and the target disparity image can be obtained by multiplying the increased first disparity image and the weight feature image element by element.

[0059] In an optional embodiment, if the resolution of the first disparity image is lower than the resolution corresponding to the weighted feature image, the resolution of the first disparity image can be increased to the resolution corresponding to the weighted feature image by using an operator in the central processing unit, and the target disparity image can be obtained by multiplying the increased first disparity image and the weighted feature image element by element.

[0060] In summary, processing the binocular images (left and right images) using a baseline disparity network yields a low-resolution initial disparity image. A first convolutional layer is then used to optimize the initial disparity image, improving its resolution and generating a higher-resolution first disparity image. Upsampling and thinning of the low-resolution disparity image compensates for the accuracy loss caused by previous quantization and attempts to recover more details. The first disparity image is then fused with a weighted feature image to obtain the target disparity image, further improving the disparity resolution. The weighted feature image, derived from feature extraction of the target image by the baseline disparity network, carries additional information about image structure and texture. By multiplying it with the first disparity image, the weighted feature image helps correct and refine the disparity estimate, resulting in a disparity map with higher accuracy and resolution. The fusion process ensures that disparity calculation utilizes information from the high-resolution image while also considering the coarse estimate from the low-resolution disparity map, balancing computational efficiency and output quality. This application specifically overcomes the parallax accuracy problem caused by quantization and dequantization in the prior art. Through multi-stage resolution enhancement and feature fusion, it improves the accuracy of the final parallax image and provides higher quality 3D information for downstream tasks.

[0061] Optionally, in the disparity processing method provided in the embodiments of this application, after performing disparity optimization on the initial disparity image through the first convolutional layer to obtain the first disparity image, the method further includes: transmitting the first disparity image and the weight feature image to the central processing unit of the target device, wherein the reference disparity network and the first convolutional layer are deployed in the neural network acceleration processor of the target device.

[0062] In an optional embodiment, the chip used in this application includes a neural network acceleration processor (such as a BPU) and a central processing unit (CPU). The optimized first disparity image and weighted feature image are transmitted from the neural network acceleration processor (such as the BPU) to the CPU of the target device so that post-processing steps, such as precision conversion and data fusion, can be performed on the CPU. For example, data transmission can be achieved through a high-speed bus or data interface inside the device.

[0063] Deploying complex neural network computations in neural network acceleration processors (such as BPUs) can fully utilize their high-performance parallel computing capabilities to accelerate the forward inference process; while allocating precision conversion and data fusion post-processing tasks to the CPU can significantly improve processing speed while ensuring the quality of parallax images, thereby better meeting real-time requirements.

[0064] Optionally, in the disparity processing method provided in this application embodiment, calculating the first disparity image and the weighted feature image to obtain the target disparity image includes: in the central processing unit, converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively; copying each data in the first floating-point data to a vector of target bit length using a first vector instruction value to obtain processed first floating-point data; and calculating the processed first floating-point data and second floating-point data using a second vector instruction value to obtain the target disparity image.

[0065] In an optional embodiment, the first disparity image and the weighted feature image are converted from integer data (such as int16 or int8) to floating-point data (i.e., first floating-point data and second floating-point data). This conversion is crucial because floating-point numbers can represent a wider range and higher precision values, which is essential for subsequent calculations (such as weighted averaging), especially when high-precision depth information is required.

[0066] Then, after conversion to floating-point type, each data point in the first disparity image is copied into a vector register using a first vector instruction (such as VDUP or FMOV in the NEON instruction set), filling the vector with the target bit length, thus forming a vector of the same dimension as the second floating-point data. For example, the first floating-point data is converted to 1 using the first vector instruction value. 9 H W's floating-point data (i.e., the first floating-point data after the above processing).

[0067] After obtaining the processed first floating-point data, a second vector instruction (such as VFMA or FMUL+VADD in NEON) is used to perform element-wise multiplication and summation on the processed first and second floating-point data to generate 1. 1 H W's target parallax image.

[0068] By utilizing the NEON instruction set on the CPU side to perform data type conversion, vectorization processing, and efficient computation on disparity images and weighted feature images, the accuracy and speed of disparity image generation are improved.

[0069] Optionally, in the disparity processing method provided in this application embodiment, calculating the processed first floating-point data and second floating-point data using the second vector instruction value to obtain the target disparity image includes: performing element-wise multiplication of the processed first floating-point data and second floating-point data to obtain a first calculation result; adding the target dimension data in the first calculation result to obtain a second calculation result; and inverse quantizing the second calculation result to obtain the target disparity image.

[0070] In an optional embodiment, calculating the processed first and second floating-point data using the second vector instruction value includes the following steps: performing element-wise multiplication of the processed first and second floating-point data using the VFMA or FMUL instruction in the NEON instruction set. This operation essentially multiplies each disparity value with its corresponding weight to reflect the importance of different disparity values. After the multiplication operation, the first calculation result, i.e., 1, is obtained. 9 H The intermediate result vector of W contains a corresponding 9-channel weighted product information for each pixel. The data in the target dimension (i.e., the second dimension) of the first calculation result are summed to obtain 1. 1 H The second calculation result for W is then processed. Finally, this second calculation result is dequantized, converting the floating-point data back to an integer format of int16 or int8. After dequantization, the final target disparity image is obtained.

[0071] In an optional embodiment, after the central processing unit receives Disp_low_unfold (i.e., the first disparity image) and weight_unfold (i.e., the weight feature image), it converts the int16 type data to floating-point type data at the CPU processor level. A single scalar is copied to Disp_low_unfold using the neon instruction, and floating-point data is loaded into a register for weight_unfold using the neon instruction, thus establishing a dimension correspondence between Disp_low_unfold and weight_unfold. The results are then multiplied element-wise using the noen instruction, and the second dimension is added dimension-wise. Finally, the results are multiplied by their respective quantization coefficients to obtain the final disparity Disp_final_mod (i.e., the target disparity image mentioned above).

[0072] Performing the above calculations on the CPU effectively utilizes the CPU's general computing power while reducing the burden on the BPU processor. Then, the efficient parallel computing capabilities of the NEON instruction set accelerate the processing speed of element-wise multiplication and summation, thereby improving the real-time performance of the overall algorithm.

[0073] Optionally, in the disparity processing method provided in this application embodiment, converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively includes: obtaining a first quantization coefficient of the first disparity image and a second quantization coefficient corresponding to the weighted feature image, wherein the first quantization coefficient is obtained when processing a binocular image of a first resolution through a reference disparity network, and the second quantization coefficient is obtained when processing a target image of the binocular image through a reference disparity network; calculating the first disparity image and the first quantization coefficient to obtain the first floating-point data; and calculating the weighted feature image and the second quantization coefficient to obtain the second floating-point data.

[0074] In an optional embodiment, when the initial disparity image is obtained by processing the binocular image through the reference disparity network, the reference disparity network synchronously outputs a first quantization coefficient. Similarly, a second quantization coefficient is obtained when the reference disparity network processes the target image in the binocular image. The first disparity image is multiplied by the first quantization coefficient and then converted into floating-point data (i.e., first floating-point data). This operation ensures that the disparity value retains sufficient dynamic range and precision in subsequent calculations, avoiding information loss due to data type limitations. The weight feature image is multiplied by the second quantization coefficient and also converted into floating-point data (i.e., second floating-point data). The converted weight feature image can more accurately reflect the weight information of each pixel.

[0075] The first quantization coefficients output by the baseline disparity network can be used to scale and shift the first disparity image, converting it into a range more suitable for floating-point operations. Floating-point data has a wider dynamic range and higher precision than integer data, which helps retain more subtle disparity information during calculation, avoiding data truncation or overflow during quantization, thus ensuring the accuracy of disparity calculation. Converting the weight feature image into floating-point data using the second quantization coefficients improves the accuracy of the weight information, thereby effectively enhancing the accuracy of the disparity results.

[0076] Optionally, in the disparity processing method provided in the embodiments of this application, the method further includes: extracting features from the target image through the backbone network in the reference disparity network to obtain a first feature vector; fusing features from the first feature vector through the feature pyramid network in the reference disparity network to obtain a second feature vector; performing convolution calculation and upsampling processing on the second feature vector to obtain a third feature vector; and obtaining a weighted feature image based on the third feature vector.

[0077] In an optional embodiment, features of the target image (e.g., the left image in a binocular image) are extracted using a backbone network in a benchmark parallax network (such as MobileNet V2), outputting a first feature vector. The backbone network can capture important features of the target image, such as texture, edges, and color, providing a foundation for subsequent depth calculations.

[0078] The first feature vector is fused using a Feature Pyramid Network (FPN) to obtain the second feature vector. FPN is a multi-scale feature fusion mechanism that integrates feature information from different levels to accommodate objects of varying sizes and details. The feature fusion process enhances the network's ability to recognize objects at different scales, ensuring that the second feature vector contains multi-level feature information from coarse to fine.

[0079] The second feature vector is subjected to convolution and upsampling to generate the third feature vector. Convolution further refines the features, while upsampling brings the feature vector's resolution close to the original image, facilitating the generation of a weighted feature image with a size matching the original image. Finally, softmax is applied to the third feature vector to obtain the weighted feature image.

[0080] In an alternative embodiment, such as Figure 3 The schematic diagram shows how a baseline disparity network processes the left and right images in a binocular imagery. Features are extracted from the left and right images using a backbone network, and then fused using a feature pyramid network to obtain image 1. 64 H / 4 The W / 4 features are used to calculate the correlation between the extracted left and right image features, resulting in the Cost volume. Then, cost aggregation is performed on the cost volume to obtain 1. 36 H / 4 The features of W / 4 are then aggregated along the second dimension of the cost, softmax is applied, and the summation is performed to obtain 1. 1 H / 4 The initial disparity Disp_low is set to W / 4; simultaneously, 1 is extracted from the left image. 64 H / 4 The W / 4 feature vector is convolved twice and upsampled to obtain the original resolution (i.e., the third feature vector mentioned above), with a resolution of 1. 9 H W.

[0081] By combining the backbone network with the feature pyramid network, the weighted feature image is ensured to contain multi-level feature information, which helps to improve the robustness and accuracy of disparity calculation.

[0082] Optionally, in the disparity processing method provided in this application embodiment, calculating the first disparity image and the weighted feature image to obtain the target disparity image includes: processing the first disparity image to obtain a second disparity image, wherein the resolution of the second disparity image is higher than the resolution of the first disparity image; and calculating the second disparity image and the weighted feature image to obtain the target disparity image.

[0083] In an optional embodiment, the steps of calculating the first disparity image and the weight feature image can also be performed directly in the neural network acceleration processor. First, the first disparity image is processed by using a specific algorithm or operator (such as bilinear interpolation, nearest neighbor interpolation, or convolutional upsampling) to increase its resolution, resulting in a second disparity image with a higher resolution. For example, 1 9 H The second disparity image of W. Comparing the second disparity image with the weighted feature image (e.g., 1) 9 H W) Further computation is performed in the neural network accelerator, generating the final target disparity image through pixel-by-pixel multiplication and dimension summation. The introduction of weighted feature images enables weighted adjustment of disparity values ​​based on the feature importance of each pixel, thereby improving the accuracy and quality of the disparity image.

[0084] In an optional embodiment, the target difference between the first disparity image and the desired resolution is determined. Based on the target difference, the first disparity image is upsampled through a convolutional layer to obtain a second disparity image corresponding to the desired resolution. Then, the target disparity image is generated through pixel-wise multiplication and dimension summation operations.

[0085] By integrating resolution enhancement and optimized computing into the accelerated processor, data transfer latency is reduced and the real-time performance of computing is improved.

[0086] Optionally, in the disparity processing method provided in this application embodiment, processing the first disparity image to obtain the second disparity image includes: calculating the first disparity image through a second convolutional layer to obtain a third calculation result, wherein the resolution of the third calculation result is the same as the resolution of the first disparity image; reshaping and sorting the third calculation result to obtain a fourth calculation result, wherein the resolution of the fourth calculation result is higher than the resolution of the third calculation result; calculating the fourth calculation result through a third convolutional layer to obtain a fifth calculation result, wherein the resolution of the fifth calculation result is the same as the resolution of the fourth calculation result; and reshaping and sorting the fifth calculation result to obtain the second disparity image.

[0087] In an optional embodiment, the first parallax image can be processed through the following steps to obtain 1. 9 H The second disparity image W (i.e., high resolution): The third calculation result is obtained by calculating the first disparity image through the second convolutional layer. For example, Disp_low_unfold (i.e., the first disparity image mentioned above) is calculated using a fixed weight of 9. Unwinding a 36-dimensional convolution is similar to copying the second dimension four times, resulting in 1. 36 H / 4 The W / 4 resolution result (i.e., the third calculation result mentioned above, with the same resolution as the first disparity image) shows that convolution calculation can further refine the information of the disparity image, remove noise through filtering operations, and enhance the expression of image features.

[0088] Then, the third calculation result is reshaped and sorted to generate the fourth calculation result. For example, performing reshape, permute, and reshape operations on the third calculation result yields a resolution of 1. 9 H / 2 The intermediate result of W / 2 (i.e., the fourth calculation result mentioned above, with a higher resolution than the first disparity image). It's important to note that `reshape` is used to change the shape (i.e., dimensions) of data without altering its values ​​or content. For example, `reshape` is used to convert one-dimensional data into multi-dimensional data, or to adjust the dimensionality of multi-dimensional data to fit specific model or algorithm requirements. `permute` is used to change the dimensional order of a tensor. For instance, the `permute` operation allows a user to specify a new dimensional order, rearranging the dimensions of the input data in that order.

[0089] After obtaining the fourth calculation result, the third convolutional layer calculates the fourth calculation result to obtain the fifth calculation result. For example, the fourth calculation result is processed with a fixed weight of 9. Unwinding a 36-fold convolution yields a resolution of 1. 36 H / 2 The intermediate result of W / 2 (i.e., the fifth calculation result mentioned above, with the same resolution as the fourth calculation result) is then reshaped and sorted to obtain the second disparity image. For example, reshape, permute, and reshape operations are performed on the fifth calculation result to obtain a resolution of 1. 9 H W's Disp_mod_unfold (i.e., the second parallax image mentioned above).

[0090] In an optional embodiment, reshaping and sorting the third calculation result to obtain the fourth calculation result includes: reshaping the third calculation result to adjust the dimensions of the third calculation result, rearranging the adjusted third calculation result to change the dimensional order of the adjusted third calculation result, and finally, reshaping the result again to obtain the fourth calculation result.

[0091] By performing operations such as convolution, reshaping, and permute, the resolution of the parallax image is effectively improved, and the feature representation is optimized, ensuring a dual improvement in image quality and computational efficiency.

[0092] In an alternative embodiment, the following can be employed: Figure 4 The diagram shown illustrates how to obtain high-resolution disparity results. The input is Disp_low (the initial disparity image output by the baseline disparity network), using a fixed-weight 3x3 matrix. 3. The convolution unfolds Disp_low to obtain Disp_low_unfold (i.e., the first disparity image), with a resolution of 1. 9 H / 4 W / 4; then, through a fixed weight of 9 A 36-dimensional convolution is performed on Disp_low_unfold, which increases the dimension of Disp_low_unfold by a factor of four, resulting in a 1-dimensional convolution. 36 H / 4 W / 4 resolution results; for 1 36 H / 4 Perform reshape, permute, reshape operations on the W / 4 resolution result to obtain a resolution of 1. 9 H / 2 intermediate results of W / 2; for 1 9 H / 2 The intermediate result of W / 2 is weighted by a fixed weight of 9. Unwinding a 36-fold convolution yields a resolution of 1. 36 H / 2 The intermediate result of W / 2, for 1 36 H / 2 Perform reshape, permute, reshape operations on the intermediate result of W / 2 to obtain a resolution of 1. 9 H The Disp_mod_unfold of W (i.e., the second disparity image) is obtained by multiplying Disp_mod_unfold and the weighted feature image, and then summing the results to obtain the final disparity Disp_final_mod (i.e., the target disparity image).

[0093] In an optional embodiment, the first, second, and third convolutional layers, along with the baseline disparity network, can be considered as a single target model. The target model is then trained and fine-tuned on the chip's neural network accelerator. The trained and fine-tuned target model is then used to process the binocular images, thereby obtaining a high-resolution disparity image. It should be noted that during training and fine-tuning, the weights of the first, second, and third convolutional layers remain fixed; the main adjustment is to the parameters of the baseline disparity network.

[0094] During training, the initial disparity results can be converted to a resolution of 1 using the first convolutional layer, the second convolutional layer, the third convolutional layer, and reshape and permute operations on the neural network accelerator processor (BPU processor). 9 H The Disp_mod_unfold of W is then calculated. The CPU processor multiplies the output Disp_mod_unfold and weight_mod, and then sums them to obtain the final disparity Disp_final_mod, for example, as shown below. Figure 5 As shown, the inputs are int16 values ​​of `Disp_mod_unfold` and `weight_mod`. On the CPU, the quantization coefficients of `Disp_unfold_mod` are obtained, and the int16 values ​​are converted to floating-point values. Similarly, the quantization coefficients of `weight_unfold` are obtained on the CPU, and the int16 values ​​are converted to floating-point values. These floating-point data are multiplied together, summed over the feature dimensions, and then multiplied by the two quantization coefficients to obtain the final disparity `Disp_final_mod`.

[0095] In subsequent applications, some of the disparity calculations that would normally run on the BPU processor are now executed on the CPU processor. This not only effectively reduces post-processing time on the CPU side but also significantly reduces inference time on the BPU side. Specific operations are as follows: Figure 6As shown, firstly, `Disp_low_unfold` and `weight_unfold` are input, and the quantization coefficients are obtained at the CPU processor, converting the int16 type data to floating-point type data. A single scalar is copied to `Disp_low_unfold` using the `neon` instruction, and `weight_unfold` is loaded into a register using the same instruction, thus establishing a dimension correspondence between `Disp_low_unfold` and `weight_unfold`. Then, the results are multiplied element-wise using the `noen` instruction, and the first dimension is added dimension-wise. Finally, the result is multiplied by its respective quantization coefficient to obtain the final disparity `Disp_final_mod`. Through these operations, the disparity processing latency is significantly reduced without affecting accuracy.

[0096] In an optional embodiment, the comparison results of the prior art and the parallax processing method provided in the embodiments of this application are shown in Table 1.

[0097] Table 1

[0098]

[0099] The aforementioned test metric, EPE (End-Point Error), is a measure of the error between predicted and true disparity. Its core idea is to compare the difference between the predicted and true values ​​pixel by pixel. The specific formula is shown below:

[0100] EPE=Mean(Sum(Abs(Pred i -Gt i )))

[0101] Where Predi and Gti are the disparity prediction result and the true disparity value, respectively, and i is the sorting order of pixels with true values; Abs is the absolute value symbol, Sum is the summation symbol, and Mean is the mean symbol. The smaller the EPE, the better the effect.

[0102] As shown in Table 1 above, the EPE value of the existing technology is 5.371, indicating that it has certain limitations in high-precision disparity calculation and may not be able to provide sufficiently accurate depth information in complex scenes. By optimizing the disparity calculation process, including but not limited to convolution calculation, feature extraction, reshaping, sorting, and specific post-processing strategies, the method of this application embodiment significantly reduces the EPE value to 0.6873. Under the same input and output conditions, the prediction of disparity is more accurate, and the detail expression is more refined, especially in key features such as edges and textures.

[0103] Regarding the time consumption of inverse quantization, a comparison between the embodiments of this application and the prior art is analyzed from the perspective of total time consumption, as shown in Table 2 below:

[0104] Table 2

[0105]

[0106] As shown in Table 2, in the prior art, the BPU processing time is 48 milliseconds, while the CPU processing time includes 22 milliseconds for inversion and 18 milliseconds for post-processing, totaling 40 milliseconds. Therefore, the total time for the entire process is 88 milliseconds. In contrast, the processing time on the BPU in this embodiment is reduced to 38 milliseconds, while the processing time on the CPU is significantly shortened to 15 milliseconds. Therefore, the overall time is only 53 milliseconds, saving approximately 39% of the time compared to the prior art. The time required for inversion is significantly reduced through the effective utilization of hardware resources (such as using NEON instructions for acceleration).

[0107] The disparity processing method provided in this application embodiment processes a binocular image through a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image; the initial disparity image is then optimized through a first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; a target disparity image is obtained by calculating the first disparity image and a weighted feature image, wherein the weighted feature image is obtained by processing the target image in the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image, thus solving the technical problem of poor accuracy of disparity values ​​in related technologies.

[0108] In this application, firstly, the binocular images (left and right images) are processed using a baseline disparity network to obtain a low-resolution initial disparity image. A first convolutional layer is then used to perform disparity optimization on the obtained initial disparity image, improving its resolution and generating a higher-resolution first disparity image. The low-resolution disparity image is upsampled and refined to compensate for the accuracy loss caused by previous quantization and to attempt to recover more details. The first disparity image is then fused with a weighted feature image to obtain the target disparity image, further improving the disparity resolution. The weighted feature image, derived from feature extraction of the target image by the baseline disparity network, carries additional information about image structure and texture. By multiplying it with the first disparity image, the weighted feature image helps to correct and refine the disparity estimate, thereby generating a disparity map with higher accuracy and resolution. The fusion process ensures that the disparity calculation utilizes information from the high-resolution image while also considering the coarse estimate from the low-resolution disparity map, balancing computational efficiency and output quality. This application specifically overcomes the parallax accuracy problem caused by quantization and dequantization in the prior art. Through multi-stage resolution enhancement and feature fusion, it improves the accuracy of the final parallax image and provides higher quality 3D information for downstream tasks.

[0109] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0110] Example 2

[0111] This application also provides a parallax processing system. It should be noted that the parallax processing system of this application can be used to execute the parallax processing method provided in this application. The parallax processing system provided in this application is described below.

[0112] According to embodiments of this application, a system for implementing the above-described parallax processing method is also provided, such as... Figure 7 As shown, the system includes:

[0113] A neural network accelerator processor is configured with a baseline disparity network and a first convolutional layer. The baseline disparity network processes the binocular images to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than that of the binocular images. The first convolutional layer performs disparity optimization on the initial disparity image to obtain a first disparity image, wherein the resolution of the first disparity image is higher than that of the initial disparity image.

[0114] The central processing unit (CPU) calculates the first disparity image and the weighted feature image to obtain the target disparity image. The weighted feature image is obtained by processing the target image of the binocular image through the reference disparity network. The resolution of the target disparity image is higher than that of the first disparity image.

[0115] Optionally, such as Figure 7 As shown, the disparity processing system consists of a neural network accelerator (BPU) and a central processing unit (CPU). The BPU is responsible for running the baseline disparity network and the first convolutional layer, primarily undertaking the computational tasks of the deep learning model, especially the convolutional operations used for disparity calculation. The CPU is responsible for post-processing tasks, including but not limited to inverse quantization, disparity adjustment, and specific mathematical operations to further optimize the disparity image and adapt to the needs of high-precision applications.

[0116] In an optional embodiment, the input binocular images (i.e., the left and right images) are processed by a baseline disparity network to obtain a low-resolution initial disparity image. The initial disparity image is then optimized using a first convolutional layer, transforming the low-resolution initial disparity image into a higher-resolution first disparity image, Disp_low_unfold. The first disparity image and the weight feature image Disp_low_unfold are processed on the CPU; for example, quantization coefficients are obtained at the CPU processor, and int16 type data is converted to floating-point type data. A single scalar is copied to Disp_low_unfold using the neon instruction, and floating-point data is loaded into a register for weight_unfold using the neon instruction, thus establishing a dimensional correspondence between Disp_low_unfold and weight_unfold. The results are then multiplied element-wise using the noen instruction, and the first dimension is added dimension-wise. Finally, the result is multiplied by its respective quantization coefficient to obtain the final disparity image, Disp_final_mod (i.e., the target disparity image).

[0117] By leveraging the collaborative capabilities of the BPU and CPU, the respective strengths of each are fully utilized. The BPU's efficient parallel computing power is used to quickly generate preliminary disparity images, while the CPU's flexibility and high-precision processing ensure the quality and applicability of the final disparity image. In assisted driving, real-time depth information is crucial for safe driving. This application can provide fast and accurate disparity maps, supporting the vehicle's real-time environmental perception and decision-making, and improving its navigation and obstacle avoidance capabilities.

[0118] Example 3

[0119] This application also provides a parallax processing device. It should be noted that the parallax processing device of this application can be used to execute the parallax processing method provided in this application. The parallax processing device provided in this application is described below.

[0120] According to an embodiment of this application, an apparatus for implementing the above-described parallax processing method is also provided, such as... Figure 8 As shown, the device includes a processing unit 801, an optimization unit 802, and a computing unit 803.

[0121] The processing unit 801 is used to process the binocular image through a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image.

[0122] The optimization unit 802 is used to perform disparity optimization on the initial disparity image through the first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image;

[0123] The calculation unit 803 is used to calculate the first disparity image and the weighted feature image to obtain the target disparity image. The weighted feature image is obtained by processing the target image in the binocular image through the reference disparity network. The resolution of the target disparity image is higher than that of the first disparity image.

[0124] The disparity processing apparatus provided in this application embodiment processes a binocular image through a reference disparity network by a processing unit 801 to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image; an optimization unit 802 performs disparity optimization on the initial disparity image through a first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; and a calculation unit 803 calculates the first disparity image and a weighted feature image to obtain a target disparity image, wherein the weighted feature image is obtained by processing the target image in the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image, thus solving the technical problem of poor accuracy of disparity values ​​in related technologies.

[0125] In this application, firstly, the binocular images (left and right images) are processed using a baseline disparity network to obtain a low-resolution initial disparity image. A first convolutional layer is then used to perform disparity optimization on the obtained initial disparity image, improving its resolution and generating a higher-resolution first disparity image. The low-resolution disparity image is upsampled and refined to compensate for the accuracy loss caused by previous quantization and to attempt to recover more details. The first disparity image is then fused with a weighted feature image to obtain the target disparity image, further improving the disparity resolution. The weighted feature image, derived from feature extraction of the target image by the baseline disparity network, carries additional information about image structure and texture. By multiplying it with the first disparity image, the weighted feature image helps to correct and refine the disparity estimate, thereby generating a disparity map with higher accuracy and resolution. The fusion process ensures that the disparity calculation utilizes information from the high-resolution image while also considering the coarse estimate from the low-resolution disparity map, balancing computational efficiency and output quality. This application specifically overcomes the parallax accuracy problem caused by quantization and dequantization in the prior art. Through multi-stage resolution enhancement and feature fusion, it improves the accuracy of the final parallax image and provides higher quality 3D information for downstream tasks.

[0126] Optionally, in the disparity processing apparatus provided in the embodiments of this application, the apparatus further includes: a transmission unit, used to perform disparity optimization on the initial disparity image through the first convolutional layer to obtain the first disparity image, and then transmit the first disparity image and the weight feature image to the central processing unit of the target device, wherein the reference disparity network and the first convolutional layer are deployed in the neural network acceleration processor of the target device.

[0127] Optionally, in the disparity processing apparatus provided in this application embodiment, the calculation unit includes: a conversion subunit, configured to convert the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively in the central processing unit; a copying subunit, configured to copy each data in the first floating-point data to a vector of a target number of bits using a first vector instruction value to obtain processed first floating-point data; and a first calculation subunit, configured to perform calculations on the processed first floating-point data and second floating-point data using a second vector instruction value to obtain a target disparity image.

[0128] Optionally, in the disparity processing apparatus provided in the embodiments of this application, the first calculation subunit includes: a first processing module, used to perform element-wise multiplication of the processed first floating-point data and second floating-point data to obtain a first calculation result; a second processing module, used to perform addition processing on the target dimension data in the first calculation result to obtain a second calculation result; and a quantization module, used to perform inverse quantization on the second calculation result to obtain a target disparity image.

[0129] Optionally, in the disparity processing apparatus provided in this application embodiment, the conversion subunit includes: an acquisition module, configured to acquire a first quantization coefficient of a first disparity image and a second quantization coefficient corresponding to a weighted feature image, wherein the first quantization coefficient is obtained when processing a binocular image of a first resolution through a reference disparity network, and the second quantization coefficient is obtained when processing a target image of the binocular image through a reference disparity network; a first calculation module, configured to calculate the first disparity image and the first quantization coefficient to obtain first floating-point data; and a second calculation module, configured to calculate the weighted feature image and the second quantization coefficient to obtain second floating-point data.

[0130] Optionally, in the disparity processing apparatus provided in this application embodiment, the apparatus further includes: an extraction subunit, used to extract features from the target image through the backbone network in the reference disparity network to obtain a first feature vector; a fusion subunit, used to fuse the first feature vector through the feature pyramid network in the reference disparity network to obtain a second feature vector; a first processing subunit, used to perform convolution calculation and upsampling processing on the second feature vector to obtain a third feature vector; and a second calculation subunit, used to obtain a weighted feature image based on the third feature vector.

[0131] Optionally, in the disparity processing apparatus provided in this application embodiment, the calculation unit includes: a second processing subunit, used to process the first disparity image to obtain a second disparity image, wherein the resolution of the second disparity image is higher than the resolution of the first disparity image; and a third calculation subunit, used to calculate the second disparity image and the weighted feature image to obtain a target disparity image.

[0132] Optionally, in the disparity processing apparatus provided in this application embodiment, the second processing subunit includes: a third calculation module, configured to calculate the first disparity image through a second convolutional layer to obtain a third calculation result, wherein the resolution of the third calculation result is the same as the resolution of the first disparity image; a third processing module, configured to reshape and sort the third calculation result to obtain a fourth calculation result, wherein the resolution of the fourth calculation result is higher than the resolution of the third calculation result; a fourth calculation module, configured to calculate the fourth calculation result through a third convolutional layer to obtain a fifth calculation result, wherein the resolution of the fifth calculation result is the same as the resolution of the fourth calculation result; and a fourth processing module, configured to reshape and sort the fifth calculation result to obtain a second disparity image.

[0133] It should be noted that the processing unit 801, optimization unit 802, and calculation unit 803 mentioned above correspond to steps S201 to S203 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0134] Example 4

[0135] Embodiments of this application may provide an electronic device. Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device may include: one or more ( Figure 9 (Only one is shown) processor 902, memory 904, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0136] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] The processor can access information and application programs stored in memory via a transmission device to execute the following steps: processing the binocular images through a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular images; performing disparity optimization on the initial disparity image through a first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image; calculating the first disparity image and a weighted feature image to obtain a target disparity image, wherein the weighted feature image is obtained by processing the target image in the binocular images through the reference disparity network, and the resolution of the target disparity image is higher than the resolution of the first disparity image.

[0138] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: after performing disparity optimization on the initial disparity image through the first convolutional layer to obtain the first disparity image, the method further includes: transmitting the first disparity image and the weight feature image to the central processing unit of the target device, wherein the reference disparity network and the first convolutional layer are deployed in the neural network acceleration processor of the target device.

[0139] The processor can access information and applications stored in memory via a transmission device to perform the following steps: calculating the first disparity image and the weighted feature image to obtain the target disparity image, including: in the central processing unit, converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively; copying each data point in the first floating-point data to a vector of target bit length using a first vector instruction value to obtain processed first floating-point data; and calculating the processed first floating-point data and second floating-point data using a second vector instruction value to obtain the target disparity image.

[0140] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: calculating the processed first floating-point data and second floating-point data using the second vector instruction value to obtain the target disparity image includes: performing element-wise multiplication of the processed first floating-point data and second floating-point data to obtain a first calculation result; adding the target dimension data in the first calculation result to obtain a second calculation result; and inverse quantizing the second calculation result to obtain the target disparity image.

[0141] The processor can invoke information and application programs stored in the memory via a transmission device to execute the following steps: converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively, including: obtaining a first quantization coefficient of the first disparity image and a second quantization coefficient corresponding to the weighted feature image, wherein the first quantization coefficient is obtained when processing a binocular image of a first resolution through a reference disparity network, and the second quantization coefficient is obtained when processing a target image of the binocular image through a reference disparity network; calculating the first disparity image and the first quantization coefficient to obtain the first floating-point data; and calculating the weighted feature image and the second quantization coefficient to obtain the second floating-point data.

[0142] The processor can access information and applications stored in the memory via a transmission device to execute the following steps: The method further includes: extracting features from the target image through the backbone network in the benchmark disparity network to obtain a first feature vector; fusing the first feature vector through the feature pyramid network in the benchmark disparity network to obtain a second feature vector; performing convolution calculation and upsampling on the second feature vector to obtain a third feature vector; and obtaining a weighted feature image based on the third feature vector.

[0143] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: calculating the first disparity image and the weighted feature image to obtain the target disparity image includes: processing the first disparity image to obtain a second disparity image, wherein the resolution of the second disparity image is higher than the resolution of the first disparity image; calculating the second disparity image and the weighted feature image to obtain the target disparity image.

[0144] The processor can access information and application programs stored in the memory via a transmission device to execute the following steps: processing a first disparity image to obtain a second disparity image includes: calculating the first disparity image through a second convolutional layer to obtain a third calculation result, wherein the resolution of the third calculation result is the same as the resolution of the first disparity image; reshaping and sorting the third calculation result to obtain a fourth calculation result, wherein the resolution of the fourth calculation result is higher than the resolution of the third calculation result; calculating the fourth calculation result through a third convolutional layer to obtain a fifth calculation result, wherein the resolution of the fifth calculation result is the same as the resolution of the fourth calculation result; and reshaping and sorting the fifth calculation result to obtain the second disparity image.

[0145] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 9This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 9 The different configurations shown.

[0146] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0147] Example 4

[0148] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the parallax processing method provided in Embodiment 1.

[0149] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0150] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform parallax processing method steps.

[0151] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0152] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0157] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A parallax processing method, characterized in that, include: The binocular images are processed by a reference disparity network to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than that of the binocular images. The initial disparity image is optimized by a first convolutional layer to obtain a first disparity image, wherein the resolution of the first disparity image is higher than that of the initial disparity image. The first disparity image and the weighted feature image are calculated to obtain the target disparity image, wherein the weighted feature image is obtained by processing the target image in the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than that of the first disparity image.

2. The method according to claim 1, characterized in that, After performing disparity optimization on the initial disparity image through a first convolutional layer to obtain a first disparity image, the method further includes: The first disparity image and the weighted feature image are transmitted to the central processing unit of the target device, wherein the baseline disparity network and the first convolutional layer are deployed in the neural network acceleration processor of the target device.

3. The method according to claim 2, characterized in that, The target disparity image is obtained by calculating the first disparity image and the weighted feature image, including: In the central processing unit, the first disparity image and the weighted feature image are converted into first floating-point data and second floating-point data, respectively; Each data point in the first floating-point data is copied to a vector of target bit length using the first vector instruction value to obtain the processed first floating-point data; The target disparity image is obtained by calculating the processed first floating-point data and the second floating-point data using the second vector instruction value.

4. The method according to claim 3, characterized in that, The target disparity image is obtained by calculating the processed first floating-point data and second floating-point data using the second vector instruction value, including: The processed first floating-point data and the second floating-point data are multiplied element by element to obtain the first calculation result; The data of the target dimension in the first calculation result are added together to obtain the second calculation result; The second calculation result is dequantized to obtain the target disparity image.

5. The method according to claim 3, characterized in that, Converting the first disparity image and the weighted feature image into first floating-point data and second floating-point data respectively includes: A first quantization coefficient of the first disparity image and a second quantization coefficient corresponding to the weighted feature image are obtained, wherein the first quantization coefficient is obtained when the binocular image of the first resolution is processed by the reference disparity network, and the second quantization coefficient is obtained when the target image of the binocular image is processed by the reference disparity network. The first floating-point data is obtained by calculating the first disparity image and the first quantization coefficient; The second floating-point data is obtained by calculating the weighted feature image and the second quantization coefficient.

6. The method according to claim 1, characterized in that, The method further includes: The target image is feature extracted using the backbone network in the baseline disparity network to obtain a first feature vector. The first feature vector is fused using the feature pyramid network in the baseline disparity network to obtain the second feature vector; The second feature vector is convolved and upsampled to obtain the third feature vector; The weighted feature image is obtained based on the third feature vector.

7. The method according to claim 1, characterized in that, The target disparity image is obtained by calculating the first disparity image and the weighted feature image, including: The first parallax image is processed to obtain a second parallax image, wherein the resolution of the second parallax image is higher than that of the first parallax image; The target disparity image is obtained by calculating the second disparity image and the weighted feature image.

8. The method according to claim 7, characterized in that, Processing the first disparity image to obtain the second disparity image includes: The first disparity image is calculated by the second convolutional layer to obtain a third calculation result, wherein the resolution of the third calculation result is the same as the resolution of the first disparity image. The third calculation result is reshaped and sorted to obtain a fourth calculation result, wherein the resolution of the fourth calculation result is higher than that of the third calculation result; The fourth calculation result is calculated by the third convolutional layer to obtain the fifth calculation result, wherein the resolution of the fifth calculation result is the same as the resolution of the fourth calculation result; The fifth calculation result is reshaped and sorted to obtain the second disparity image.

9. A parallax processing system, characterized in that, include: A neural network acceleration processor is provided, wherein the neural network acceleration processor is deployed with a reference disparity network and a first convolutional layer. The reference disparity network processes the binocular image to obtain an initial disparity image, wherein the resolution of the initial disparity image is lower than the resolution of the binocular image. The first convolutional layer performs disparity optimization on the initial disparity image to obtain a first disparity image, wherein the resolution of the first disparity image is higher than the resolution of the initial disparity image. A central processing unit (CPU) calculates a target disparity image by processing the first disparity image and the weighted feature image, wherein the weighted feature image is obtained by processing the target image of the binocular image through the reference disparity network, and the resolution of the target disparity image is higher than that of the first disparity image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the parallax processing method according to any one of claims 1 to 8.

Images