Image processing method and apparatus

Abstract

Embodiments of the present specification provide an image processing method and device, wherein the image processing method comprises: acquiring a first view and a second view, and inputting the first view and the second view into an image processing model, wherein the image processing model comprises a feature extraction module and a matrix construction module; extracting, by using the feature extraction module, a first view feature corresponding to the first view and a second view feature corresponding to the second view; performing, by using the matrix construction module, matching cost calculation on the first view feature and the second view feature within a preset disparity search range, generating an initial cost matrix according to a calculation result; performing cost aggregation on a plurality of matching costs contained in the initial cost matrix, obtaining an aggregated cost matrix, and performing pixel disparity processing on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of image processing technology, and particularly to image processing methods and apparatus. Background Technology

[0002] Stereo matching is a core technology in computer vision, aiming to obtain depth information of a scene by calculating the disparity between images from left and right cameras. It is widely used in autonomous driving, robot navigation, and 3D reconstruction. Existing stereo matching technologies are mainly divided into two categories: traditional algorithms and end-to-end deep learning methods. Traditional algorithms typically follow a process of feature extraction, cost matrix construction, cost aggregation, and disparity calculation, relying on manually designed feature operators to construct the cost matrix and using strategies such as semi-global matching for optimization. End-to-end deep learning methods, on the other hand, utilize convolutional neural networks or Transformer architectures to directly regress a disparity map from the input image. Through large-scale data training, they automatically learn feature representations and matching relationships, achieving fully automated processing from feature extraction to disparity output. However, in existing technologies, when facing complex scenes with weak texture regions, repetitive texture structures, and drastic lighting changes, it is often difficult to simultaneously ensure both robustness and detail in the matching. Traditional methods tend to generate numerous ambiguous matching points in regions lacking texture information, resulting in holes or noise in the generated depth maps. Meanwhile, pure deep learning solutions often suffer from high-frequency detail loss due to network downsampling operations or regression smoothing characteristics when dealing with subtle disparity variations and object edge sharpness, making it difficult to meet the stringent requirements of sub-pixel-level disparity estimation in high-precision applications. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of this, embodiments of this specification provide an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, an image processing method is provided, comprising: A first view and a second view are obtained, and the first view and the second view are input into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module is used to extract the first view features corresponding to the first view and the second view features corresponding to the second view. The matrix construction module is used to calculate the matching cost for the first view feature and the second view feature within a preset disparity search range, and an initial cost matrix is ​​generated based on the calculation results. Cost aggregation is performed on the multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix. Then, pixel disparity processing is performed on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

[0005] According to a second aspect of the embodiments of this specification, another image processing method is provided, comprising: Multiple views are acquired and input into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module is used to extract the view features corresponding to each view. Using the matrix construction module, matching cost is calculated for the view features corresponding to each view within a preset disparity search range, and an initial cost matrix is ​​generated based on the calculation results. Cost aggregation is performed on the multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix. Then, pixel disparity processing is performed on the multiple views based on the aggregated cost matrix to obtain a target disparity map.

[0006] According to a third aspect of the embodiments of this specification, an image processing apparatus is provided, comprising: The acquisition module is configured to acquire a first view and a second view, and input the first view and the second view into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The extraction module is configured to extract first view features corresponding to the first view and second view features corresponding to the second view using the feature extraction module. The calculation module is configured to use the matrix construction module to perform matching cost calculation on the first view feature and the second view feature within a preset disparity search range, and generate an initial cost matrix based on the calculation result; The processing module is configured to perform cost aggregation on multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and to perform pixel disparity processing on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

[0007] According to a fourth aspect of the embodiments of this specification, another image processing apparatus is provided, comprising: The view acquisition module is configured to acquire multiple views and input the multiple views into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module is configured to extract view features corresponding to each view using the feature extraction module; The matrix generation module is configured to use the matrix construction module to perform matching cost calculation within a preset disparity search range for the view features corresponding to each view, and generate an initial cost matrix based on the calculation results. The cost aggregation module is configured to aggregate costs for multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and to perform pixel disparity processing on the multiple views based on the aggregated cost matrix to obtain a target disparity map.

[0008] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0009] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the image processing method described above.

[0010] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the image processing method described above.

[0011] The image processing method provided in this application constructs an image processing model containing a feature extraction module and a matrix construction module. First, a first view and a second view are acquired and input into the model. The feature extraction module extracts features from the first and second views, respectively, which possess multi-scale receptive fields and high-dimensional semantics. This overcomes the sensitivity of traditional handcrafted features to abrupt changes in illumination and weakly textured regions, significantly improving the robustness of feature representation. Subsequently, the matrix construction module calculates matching costs based on the extracted depth features within a preset disparity search range and generates an initial cost matrix. Leveraging the strong discriminative power of high-dimensional features, ambiguous points in the matching process are effectively reduced. Based on this, a cost aggregation operation is performed on multiple matching costs in the initial cost matrix to obtain an aggregated cost matrix. A path constraint mechanism is introduced to enhance the spatial consistency of disparity and suppress noise interference. Furthermore, pixel disparity processing is performed based on the aggregated cost matrix. A continuous domain fitting technique is used to refine discrete integer-pixel disparities into high-precision sub-pixel disparities, ultimately obtaining the target disparity map. This effectively solves the problems of low matching accuracy and loss of high-frequency details in existing technologies under complex environments, and achieves depth perception with both high robustness and high precision, thereby improving the system's environmental adaptability and measurement reliability in practical applications such as autonomous driving and robot navigation. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an image processing method provided in one embodiment of this specification; Figure 2 This is a flowchart of another image processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of another image processing apparatus provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0018] ResNet (Residual Network) is a deep convolutional neural network (CNN) architecture. The residual module introduces skip connections across layers, directly passing the input to the output, forming a "residual learning" mechanism. In this way, the network does not directly learn the target mapping H(x), but learns the residual function F(x) = H(x) - x, and then obtains the final output through H(x) = F(x) + x.

[0019] Transformer: By introducing a self-attention mechanism, it can track relationships in sequential data and learn the context and meaning between elements. It uses a position encoder to label data elements entering and leaving the network, and uses attention units to compute the relationships between these elements.

[0020] Semi-Global Matching (SGM) is a matching algorithm for stereo vision that estimates depth information by calculating the disparity of each pixel in an image. SGM is widely used in disparity map computation; its accuracy is affected by various factors, but it can balance computational efficiency and matching precision to some extent. The SGM algorithm constructs a disparity map by selecting the disparity of each pixel and setting a global energy function associated with the disparity map. The optimal disparity for each pixel is found by minimizing this energy function. The SGM algorithm approximates the two-dimensional optimum by using multi-directional one-dimensional cost aggregation, that is, performing one-dimensional cost aggregation on all paths around the pixel (e.g., 8 or 16 directions) and then summing all the one-dimensional cost aggregation values.

[0021] This specification provides an image processing method, and also relates to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0022] Stereo matching is a core task in computer vision, aiming to obtain depth information by calculating the disparity between images from left and right cameras. Existing stereo matching algorithms are mainly divided into traditional methods and end-to-end deep learning methods. Traditional methods typically rely on hand-designed features (such as Census transform, SAD, NCC, etc.) to construct the cost matrix. These features only focus on pixel intensity differences in local minimal receptive fields, making them extremely sensitive to sudden changes in illumination and image noise. Furthermore, they fail to extract effective global semantic context information in weakly textured areas such as white walls and glass, or in areas with repetitive textures such as fences, resulting in a large amount of ambiguity in the cost matrix, which is difficult to compensate for even with powerful aggregation algorithms. While end-to-end deep learning methods solve the problem of context information extraction, they usually require multiple downsampling or pooling operations to expand the receptive field, which can lead to the loss of high-frequency spatial details and poor performance on targets with extremely sharp edges or minimal disparity. At the same time, these methods often treat disparity calculation as a classification and regression problem, resulting in overly smoothed disparity maps at object edges, making it difficult to output fine sub-pixel level disparity, and exhibiting weak generalization ability when facing unseen real-world scenes.

[0023] In view of this, the image processing method provided in this application constructs an image processing model including a feature extraction module and a matrix construction module. First, a first view and a second view are acquired and input into the model. The feature extraction module extracts features from the first and second views, respectively, which possess multi-scale receptive fields and high-dimensional semantics. This overcomes the shortcomings of traditional handcrafted features, which are sensitive to sudden changes in illumination and weakly textured regions, significantly improving the robustness of feature representation. Subsequently, the matrix construction module calculates matching costs based on the extracted depth features within a preset disparity search range and generates an initial cost matrix. Leveraging the strong discriminative power of high-dimensional features, ambiguous points in the matching process are effectively reduced. Based on this, a cost aggregation operation is performed on multiple matching costs in the initial cost matrix to obtain an aggregated cost matrix. A path constraint mechanism is introduced to enhance the spatial consistency of disparity and suppress noise interference. Furthermore, pixel disparity processing is performed based on the aggregated cost matrix. A continuous domain fitting technique is used to refine discrete integer pixel disparities into high-precision sub-pixel disparities, ultimately obtaining the target disparity map. This effectively solves the problems of low matching accuracy and loss of high-frequency details in existing technologies under complex environments, and achieves depth perception with both high robustness and high precision, thereby improving the system's environmental adaptability and measurement reliability in practical applications such as autonomous driving and robot navigation.

[0024] See Figure 1 , Figure 1 A flowchart of an image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0025] Step S102: Obtain a first view and a second view, and input the first view and the second view into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module.

[0026] The image processing method provided in this embodiment can be applied to any scenario that requires the use of scene depth information, such as autonomous driving, robot navigation, and 3D reconstruction. This embodiment uses a robot navigation scenario as an example to illustrate the image processing method. Descriptions of image processing methods in other scenarios can be found in the same or corresponding descriptions in this embodiment, and will not be elaborated upon further here.

[0027] Specifically, the first and second views refer to a pair of images used for stereo matching, which can be understood as a left and right view after epipolar correction. The first and second views are acquired simultaneously or nearly simultaneously by image acquisition devices (such as binocular cameras, vision sensors on robots, etc.), and there is pixel displacement, i.e., parallax, between them due to different viewing angles. The image processing model is a neural network model that integrates a feature extraction module and a matrix construction module. The feature extraction module can be a neural network component used to extract high-dimensional semantic information from the original image. Its role is to replace traditional manual feature operators to obtain feature representations robust to changes in illumination and weak textures. The matrix construction module can be a component used to calculate the matching cost and construct a three-dimensional cost volume based on the extracted features. Its role is to convert the similarity in the feature space into the cost in the parallax space. The feature extraction module and the matrix construction module work together; the former provides high-quality input data for the latter, and the latter constructs the initial matching relationship based on the output of the former, together forming the transformation link from the original image to the initial cost matrix.

[0028] Based on this, a first and second view are acquired and input into the image processing model, primarily to provide a highly discriminative feature base for subsequent disparity calculation. Using the acquired first and second views, the image processing model first calls the feature extraction module to encode the two images, generating feature maps containing multi-scale receptive fields and high-dimensional semantics. These feature maps are then passed to the matrix construction module. By employing a deep learning network as the feature extraction method, the limitations of traditional handcrafted features in complex environments can be effectively overcome, providing a prerequisite for generating a highly robust initial cost matrix.

[0029] Step S104: Use the feature extraction module to extract the first view features corresponding to the first view and the second view features corresponding to the second view.

[0030] Specifically, the first-view features can refer to the multi-dimensional feature tensor extracted from the first view, which contains the semantic and contextual information of each pixel in the first view. The second-view features can refer to the corresponding multi-dimensional feature tensor extracted from the second view. The first-view and second-view features are obtained by performing forward propagation calculations on the input first and second views respectively using a weight-sharing neural network (such as ResNet, Transformer, or other convolutional neural networks). Weight sharing means that the network parameters processing the first and second views are exactly the same, ensuring that the two sets of extracted features are in the same feature space, facilitating subsequent similarity measurement. The role of the first-view and second-view features is to characterize the local and global texture, edges, and semantic structure of the image, replacing traditional pixel grayscale values ​​for matching in subsequent steps, thereby significantly improving feature discrimination in scenes with weak texture, high reflectivity, and drastic lighting changes.

[0031] For example, when the resolution of the first view and the second view is both At that time, after processing by the feature extraction module, the first view features are generated. Second view features The dimension can be ,in This refers to the number of feature channels (e.g., 64 or 128). and The height and width of the feature map (which may be the same as the original image or slightly downsampled) are given. In this high-dimensional space, even if a region of the original image is pure white and textureless, its feature vector can still exhibit differences through deep semantic information (such as object category, relative position, etc.) learned by the network, thus avoiding matching ambiguity in traditional methods. This deep feature extraction method can significantly reduce the false matching rate, laying a solid foundation for constructing a high-precision cost matrix.

[0032] Based on this, the powerful nonlinear mapping capability of deep neural networks is used to transform the original pixel signals into high-dimensional feature representations rich in semantics. This significantly improves the robustness of feature description, effectively avoiding feature failure caused by sudden changes in illumination or missing textures, making subsequent feature-based matching calculations more reliable.

[0033] Step S106: Using the matrix construction module, perform matching cost calculation for the first view feature and the second view feature within a preset disparity search range, and generate an initial cost matrix based on the calculation results.

[0034] Specifically, the preset parallax search range can refer to the maximum parallax value set according to the application scenario. The determined discrete disparity set typically covers values ​​from 0 to... All integer disparities. Matching cost can refer to a numerical value that measures the similarity or difference between a point in the first view feature and its corresponding point in the second view feature under a specific disparity. The initial cost matrix is ​​a three-dimensional data structure, typically with dimensions of 1. (or ), where each element Represents the coordinates in the first view At this point, assume the parallax is... The matching cost is calculated by taking the inner product (correlation) or distance (such as Euclidean distance or cosine distance) of the first view features and the second view features under a preset disparity. For example, cosine similarity can be used to calculate the cost. For example, the initial cost C(x, y, d) of pixel coordinates (x, y) at disparity d can be defined by the following formula (1): (1) in, and Let represent the features of the first view and the features of the second view, respectively, and d represent the disparity. Based on this, an initial cost matrix can be constructed. ;in, A smaller value indicates a higher matching degree. The purpose of the initial cost matrix is ​​to exhaustively enumerate all possible disparity hypotheses, forming a complete matching probability distribution map, providing raw data support for subsequent optimization and aggregation. The matrix construction module works in conjunction with the feature extraction module, utilizing the strong discriminative power of high-dimensional features to ensure that the generated initial cost matrix can present clear minimum points even in weak texture regions, reducing ambiguity.

[0035] Step S108: Cost aggregation is performed on the multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and pixel disparity processing is performed on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

[0036] Specifically, cost aggregation refers to spatial domain optimization of the cost values ​​in the initial cost matrix, utilizing the consistency constraints of neighboring pixels to suppress noise and fill holes in weakly textured regions. The aggregated cost matrix is ​​the optimized 3D cost data, whose internal cost value distribution is smoother and has stronger edge-preserving properties than the initial cost matrix. Cost aggregation is achieved by performing cumulative calculations under path constraints in multiple spatial directions (such as 8 or 16 directions) using the semi-global matching (SGM) algorithm or other traditional dynamic programming algorithms. Specifically, for any pixel, the algorithm accumulates the minimum path cost along different directions and introduces a penalty coefficient. and The process adapts to smooth regions and depth-discontinuous edges separately, ultimately combining the costs from all directions to obtain the aggregated total cost. Pixel disparity processing refers to the process of determining the final disparity value of each pixel based on the aggregated cost matrix. The target disparity map is the final output two-dimensional image, where the grayscale or color value of each pixel represents the disparity magnitude of that point in three-dimensional space. The target disparity map is obtained by selecting the disparity corresponding to the minimum cost value in the aggregated cost matrix using a winner-take-all (WTA) strategy, and further combining it with continuous domain fitting techniques such as parabolic interpolation for sub-pixel thinning.

[0037] For example, in obtaining the aggregation cost matrix Then, first select the one that makes Minimum integer disparity As the initial integer-pixel disparity. Then, using... and its left and right adjacent parallax ( and The cost of the pixel disparity is calculated using a parabolic fitting formula, yielding sub-pixel disparity values ​​accurate to multiple decimal places. This approach avoids disparity quantization errors and excessive edge smoothing issues caused by classification and regression in end-to-end networks. The cost aggregation module works in conjunction with the pixel disparity processing module; the former enhances global consistency of matching through multi-directional constraints, while the latter recovers high-frequency details through refined geometric fitting. Together, they achieve a balance between high robustness and high accuracy.

[0038] Based on this, the above processing can eliminate noise interference in the initial matching and achieve high-precision disparity estimation. After obtaining the initial cost matrix, multi-directional cost aggregation can be performed to enhance the reliability of the matching results. Subsequently, pixel-level filtering and sub-pixel refinement are performed based on the aggregated cost distribution. This significantly improves the sharpness of the disparity map at object edges and the overall accuracy, effectively avoiding the over-smoothing defects of traditional deep learning methods, and ultimately obtaining a high-quality target disparity map.

[0039] In summary, by introducing deep neural networks to extract robust and semantically rich multi-scale features, the matching accuracy in weak and repetitive texture regions is significantly improved. Furthermore, by abandoning the 3D convolutional aggregation layers that cause detail loss in end-to-end networks, a traditional semi-global matching algorithm is used for cost aggregation, perfectly preserving the sharp edges of objects. Then, traditional parabolic interpolation is combined for sub-pixel refinement, overcoming the bottleneck of pure deep learning in refining disparity. This organic combination of deep feature construction and traditional cost aggregation not only avoids the extremely memory-intensive 3D CNN operations, making computational resources controllable, but also allows the model to exhibit generalization capabilities far exceeding those of end-to-end networks when facing cross-scene data. Ultimately, it outputs a target disparity map with high robustness, high detail fidelity, and high sub-pixel accuracy.

[0040] In one or more embodiments of this example, obtaining the first view and the second view, and inputting the first view and the second view into the image processing model, includes: A first view is acquired through a first image acquisition device of the target robot, and a second view is acquired through a second image acquisition device of the target robot; the first view and the second view are input into an image processing model, wherein the image processing model uses a weight-shared neural network as a feature extraction module, and the feature extraction module is used to extract view features containing multi-scale receptive fields and high-dimensional semantics.

[0041] Specifically, the target robot can refer to an intelligent carrier with autonomous mobility or remote control capabilities, such as autonomous vehicles, warehousing and logistics robots, or service humanoid robots. The first and second image acquisition devices are two physically fixed camera modules mounted on the target robot and epipolar-corrected, typically corresponding to the left and right cameras in a stereo vision system, respectively. These two acquisition devices are used to synchronously capture image data of the same scene from different perspectives, thereby obtaining a first view and a second view with parallax information. Specifically, the first and second image acquisition devices can achieve microsecond-level time synchronization through hardware trigger signals to eliminate timing errors caused by robot movement or dynamic changes in the scene.

[0042] Correspondingly, the image processing model is a deep learning processing architecture deployed in the onboard computing unit of the target robot or a cloud server. A weight-sharing neural network can refer to a feature extractor containing two structurally identical and parameter-bound sub-network branches. That is, the first and second views enter two independent input channels, but these two channels share the same set of convolutional kernel weights, bias parameters, and normalized statistics. This weight-sharing mechanism ensures that for image regions with the same content but different viewpoints, the network can output completely consistent feature responses, thereby eliminating matching bias introduced by asymmetric network parameters. Specifically, the feature extraction module can adopt ResNet, Transformer, or their variants, which internally construct a pyramid-shaped feature representation system by concatenating different levels of convolutional groups or attention mechanisms. Multi-scale receptive fields can refer to the ability of neurons of different depths in the network to cover local regions of different sizes in the input image. Shallow neurons focus on pixel-level edge texture details, while deep neurons focus on the overall semantic contour of the object. High-dimensional semantics can refer to the fact that the feature vector corresponding to each pixel in the feature map not only contains color gradient information but also encodes the object category, material properties, and spatial context of that pixel.

[0043] Based on this, in a robotic scenario, a first view can be acquired using the first image acquisition device of the target robot, and a second view can be acquired using the second image acquisition device of the target robot. Then, the first and second views can be input into an image processing model, and the image processing model can use a weight-sharing neural network as a feature extraction module, so that the feature extraction module can be used to extract view features containing multi-scale receptive fields and high-dimensional semantics.

[0044] In other words, the above processing utilizes the robot's stable mechanical structure to ensure the long-term stability of the camera's intrinsic and extrinsic parameters, thereby effectively avoiding the shaking and calibration drift problems caused by handheld shooting or non-fixed equipment installation, and significantly improving the operability of the image acquisition process and the specificity of deployment scenarios.

[0045] For example, a binocular camera with a baseline distance of x centimeters can be installed at the head of the target robot. The first image acquisition device is responsible for acquiring the real-time video stream of the left area as the first view, and the second image acquisition device is responsible for acquiring the real-time video stream of the right area as the second view. This dual-channel synchronous acquisition method ensures that the left and right views strictly satisfy the epipolar constraint in spatial geometry, providing a consistent data foundation for subsequent high-precision disparity calculations. Furthermore, when a ResNet-50 with four stages is used as the weight-sharing feature extraction module, the feature map output by the first stage retains high-frequency details of the original resolution and has a small receptive field, suitable for capturing subtle texture changes; while the feature map output by the fourth stage has undergone multiple downsampling, resulting in a receptive field covering a larger area of ​​the original image, with a feature vector dimension as high as 2048 dimensions, capable of clearly distinguishing semantic categories such as road conditions, objects, and animals. By simultaneously inputting the first and second views into this shared network, the system can generate two sets of feature maps that are highly aligned in the feature space in parallel. and .

[0046] In summary, by combining the collaborative work of dual acquisition devices on the target robot with deep feature extraction using a weight-sharing neural network, end-to-end optimization from data acquisition to feature representation was achieved. Stable integration with the robot platform ensured strict registration of the first and second views in the spatiotemporal domain, eliminating interference from non-rigid deformation. Furthermore, the weight-sharing neural network architecture forced the feature mappings of the left and right views to follow the same transformation logic, resulting in natural geometric correspondence among the extracted multi-scale receptive field features. Meanwhile, high-dimensional semantic features effectively compensated for the shortcomings of traditional hand-crafted features in abstract understanding. This collaborative approach of synchronous hardware acquisition and shared software parameters not only reduced the complexity of model training and the risk of overfitting but also enabled the generated view features to possess both accuracy in local details and discriminative power in global semantics. This significantly improved the adaptability and matching accuracy of the stereo matching system in complex environments in practical applications such as autonomous driving and robot navigation.

[0047] In one or more embodiments of this example, the step of using the matrix construction module to calculate the matching cost for the first view feature and the second view feature within a preset disparity search range, and generating an initial cost matrix based on the calculation result, includes: Using the matrix construction module, the matching cost corresponding to each preset disparity within the preset disparity search range is calculated for the first view feature and the second view feature; and an initial cost matrix is ​​constructed based on the matching cost corresponding to each preset disparity.

[0048] Specifically, the preset parallax search range can refer to a set of parallax values ​​pre-defined based on the depth requirements of the target scene, typically expressed as... ,in The maximum search disparity is determined. The matching cost is used to quantify the similarity or difference between the features of the first view and the features of the second view at a specific displacement. Specifically, the matrix construction module iterates through each integer disparity value within the preset disparity search range. For any pixel in the feature map of the first view Offset its feature vector to the corresponding position in the feature map of the second view. The next pixel The feature vectors are compared. The matching cost can be calculated based on the inner product (relevance) of the feature vectors or a distance metric. For example, when using cosine similarity as the metric, the matching cost... It can be defined as 1 minus the normalized inner product of the two feature vectors; when using Euclidean distance, the matching cost is the square of the magnitude of the difference between the two feature vectors. By calculating disparity one by one, it is ensured that no potential matching points are missed in the entire search space, thus overcoming the deficiency of traditional handmade features in extracting effective context in weakly textured regions.

[0049] Accordingly, the initial cost matrix is ​​a three-dimensional tensor structure with dimensions of disparity, height, and width. The matrix is ​​constructed by stacking the matching cost distribution maps for all individual disparities calculated above along the third dimension, in ascending order of disparity values. Specifically, if the preset disparity search range includes... There are disparity values, and the height of the image feature map is [value]. Width is The size of the constructed initial cost matrix is ​​then... Each element in the matrix Stored coordinates in the first view At this point, assume the parallax is... The matching cost at that time. For example, when the maximum search disparity... When set to 192, the initial cost matrix will contain 192 channels, each corresponding to a full-image matching cost heatmap under a specific disparity. By integrating the matching costs corresponding to all preset disparities into a unified matrix structure, a structured expression of the stereo matching problem is achieved, enabling subsequent algorithms to utilize both local matching information and global disparity constraints simultaneously. This result provides a standardized input format for subsequent semi-global matching algorithms to perform cost aggregation under multi-directional path constraints, significantly improving the efficiency and accuracy of disparity calculation.

[0050] Based on this, the above processing achieves an effective combination of deep features and traditional geometric constraints. High-dimensional semantic features extracted by a weight-sharing neural network are used as input, replacing the fragile grayscale or gradient features in traditional algorithms. This allows for the sharp capture of deep semantic associations in weakly textured and highly reflective areas when calculating the matching cost corresponding to each preset disparity. Furthermore, by systematically traversing the entire preset disparity search range and stacking the calculation results into an initial cost matrix, not only is extreme spatial detail resolution preserved, avoiding the loss of small disparity information due to multiple downsampling in end-to-end networks, but a high-quality data foundation is also laid for the subsequent introduction of traditional optimization strategies such as semi-global matching (SGM). This synergistic mechanism of deep feature construction + traditional cost aggregation leverages both the strong robustness of deep learning in feature representation and the advantages of traditional algorithms in sub-pixel accuracy and edge preservation, ultimately outputting a target disparity map with high detail fidelity and high accuracy.

[0051] In one or more embodiments of this example, determining the matching cost corresponding to any preset disparity includes: The matching cost corresponding to the preset disparity is obtained by calculating the inner product of the first view feature and the second view feature under a preset disparity using the matrix construction module; or, the matching cost corresponding to the preset disparity is obtained by calculating the distance between the first view feature and the second view feature under a preset disparity using the matrix construction module.

[0052] Specifically, the inner product can refer to the dot product operation of the feature vectors of the first view and the second view at corresponding spatial positions and disparity offsets, or a similarity measure based on the dot product. This inner product is obtained by element-wise multiplying the feature vector of a pixel in the first view feature map with the feature vector of the corresponding pixel in the second view feature map, which has been shifted along the epipolar direction by a predetermined disparity distance, and then summing the results. Its function is to measure the directional consistency between two high-dimensional feature vectors. The larger the inner product value, the smaller the angle between the two feature vectors, the higher the semantic similarity, and thus the greater the probability of matching. For example, when both the first and second view features are normalized 64-dimensional vectors, calculating their inner product at a disparity of 10 pixels, if the result is close to 1, the matching degree at that location is considered extremely high; if the result is close to 0 or negative, a mismatch is determined. Through this similarity measure based on the inner product, the high-dimensional semantic information extracted by deep neural networks can be effectively utilized, maintaining high matching robustness even in areas with weak texture or varying lighting, thereby accurately obtaining the matching cost corresponding to the predetermined disparity.

[0053] Correspondingly, distance can refer to the geometric distance between the feature vectors of the first view and the second view in the feature space, specifically including L1 distance (Manhattan distance), L2 distance (Euclidean distance), or Hamming distance, etc. This distance is obtained by calculating the sum or square of the absolute values ​​of the differences between the corresponding dimensional values ​​of the two feature vectors. Its function is to quantify the difference between two features; the smaller the distance value, the closer the two features are, and the higher the degree of matching. For example, using the L1 distance calculation method, for a candidate position with a disparity of 5 pixels, the sum of the absolute values ​​of the differences between each component of the feature vector of the first view and the corresponding component of the feature vector of the second view is calculated. If the sum is 0.2, while the sum of the other candidate position is 1.5, then the former is considered a better match. By introducing a distance metric mechanism, the system can flexibly adapt to application scenarios sensitive to feature amplitude, capture subtle spatial offset differences, and thus obtain the matching cost corresponding to the preset disparity.

[0054] Based on this, the synergistic effect of technical features is achieved by supporting two different matching cost calculation modes: inner product and distance. Specifically, inner product calculation focuses on the directional consistency of feature vectors, which can fully utilize the angular information in high-dimensional semantic features, making it particularly suitable for scenarios that emphasize semantic understanding and broad contextual relationships; while distance calculation focuses on the precise alignment of feature values, which can keenly capture differences in local details, making it suitable for tasks with extremely high edge accuracy requirements. These two methods complement each other in the matrix construction module, enabling the image processing model to select the most suitable metric function based on the distribution characteristics of the actual input data or the specific application scenario requirements. On this basis, regardless of which method is used to generate the matching cost, it can serve as a high-quality input for the subsequent cost aggregation step, effectively overcoming the limitations of a single similarity metric in complex scenarios, and significantly improving the generalization ability and overall accuracy of the stereo matching algorithm in different data domains.

[0055] In one or more embodiments of this example, the step of aggregating the costs of multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix includes: The initial cost matrix contains multiple matching costs, which are aggregated under path constraints in multiple spatial directions using a semi-global matching algorithm to obtain an aggregated cost matrix to be optimized. The aggregated cost matrix to be optimized is then optimized according to a preset cost optimization strategy to obtain an aggregated cost matrix.

[0056] Specifically, the semi-global matching (SGM) algorithm is a dynamic programming-based global optimization approximation algorithm used to balance computational efficiency and matching accuracy. The core of this algorithm lies in decomposing the two-dimensional or three-dimensional global energy minimization problem into multiple one-dimensional path optimization problems. Specifically, these multiple spatial directions can refer to several scanning paths radiating outwards from the current pixel, typically including horizontal, vertical, and diagonal directions, such as 8 or 16 directions. On each path, the algorithm recursively accumulates the matching cost and introduces a smoothness constraint to penalize drastic changes in disparity.

[0057] Specifically, for any pixel and in a certain direction Front-end pixels and candidate parallax Its path cumulative cost The calculation relies on the sum of the accumulated cost of the predecessor point and the initial matching cost of the current point, plus a corresponding penalty coefficient. If the disparity change between adjacent pixels is 0 or 1 (i.e., disparity is continuous or slightly tilted), a smaller first penalty coefficient is applied. If the disparity change between adjacent pixels is greater than 1 (i.e., there are object edges with depth discontinuities), then a larger second penalty coefficient is applied. This differentiated penalty mechanism enables the algorithm to suppress parallax jitter caused by noise in smooth regions, while preserving abrupt depth changes at object edges to avoid over-smoothing. That is, the aggregation cost for any pixel p = (x, y) in direction r. The update can be performed according to the following formula (2): (2) Here, pr represents the previous pixel along the direction r. A small penalty coefficient is used to accommodate tilted surfaces (parallax variation is 1). A larger penalty coefficient is used to preserve depth discontinuities at object edges (parallax variation greater than 1). Finally, the aggregated costs from all directions (typically 8 or 16 directions) are combined to obtain the total cost matrix. .

[0058] For example, the maximum disparity search range is set to 128, and eight scanning directions (up, down, left, right, and four diagonals) are selected. When processing a white wall area in an image, since the initial cost matrix may contain random noise due to missing textures, the SGM algorithm accumulates the costs in the eight directions, allowing the correct disparity path to stand out due to the lowest accumulated cost, while the incorrectly matched noisy paths are suppressed due to smoothness penalties. Similarly, when processing object contour edges, if a pixel's true disparity jumps significantly with its neighboring pixels, the algorithm automatically identifies this as a large disparity change and applies a larger penalty coefficient. This allows for disparity discontinuities at that location, ensuring sharp and clear object boundaries. Through this cost aggregation under multi-directional path constraints, contextual information from different angles can be effectively fused, significantly reducing the uncertainty of single-directional matching, thus obtaining an aggregated cost matrix to be optimized that contains richer spatial consistency information.

[0059] Correspondingly, the preset cost optimization strategy can refer to the subsequent processing logic executed after multi-directional path aggregation to further eliminate residual noise, correct outliers, or adapt to specific scene requirements. The aggregation cost matrix to be optimized is the sum of the cumulative costs of each directional path. Although it already has strong robustness, it may still contain extreme point offsets or local optimum deviations in occluded areas, weak texture areas, or complex lighting conditions. Cost optimization strategies include, but are not limited to, cost normalization, left-right consistency check, filtering processing before sub-pixel interpolation, or confidence-based culling operations.

[0060] Specifically, cost normalization maps the aggregated total cost to a fixed numerical range (e.g., 0 to 255 or 0 to 1) to prevent numerical overflow and facilitate subsequent comparisons. Left-right consistency detection utilizes binocular geometric constraints to verify whether the optimal disparity in the left view corresponds to the same matching point in the right view. If they are inconsistent, it is determined to be occlusion or a mismatch, and the cost at that point is marked as invalid or given a large penalty value. Furthermore, weighted median filtering or guided filtering can be used to smooth and denoise the aggregated cost matrix to be optimized, further removing isolated erroneous matching points.

[0061] For example, after obtaining the aggregation cost matrix to be optimized, the system executes a left-right consistency detection strategy: for pixels in the left image... Assuming its calculated optimal disparity is Then check the corresponding position in the right figure. Is the optimal disparity also... If the deviation between the two exceeds a preset threshold (e.g., 1 pixel), the point is considered to be in an occluded area or a mismatch, and its corresponding cost value in the aggregated cost matrix is ​​set to infinity or repaired based on reliable neighboring points. By applying such preset cost optimization strategies, the data quality in the cost matrix can be further purified, eliminating false low-cost points caused by occlusion or duplicate textures, ultimately generating a high-confidence, high-consistency aggregated cost matrix, laying a solid foundation for subsequent accurate disparity calculation.

[0062] In summary, the above processing achieves decoupling and complementarity between feature representation and geometric optimization in stereo matching. First, a semi-global matching algorithm is used to perform cost aggregation under path constraints in multiple spatial directions. This transforms the scattered and noise-sensitive matching information in the initial cost matrix into a strongly spatially continuous aggregated cost matrix to be optimized. This process effectively utilizes the prior knowledge of the image's two-dimensional structure, significantly improving the robustness of matching in weak and repetitive texture regions without relying on expensive 3D convolution operations. Based on this, a pre-defined cost optimization strategy is used to perform secondary cleaning and correction on the aggregated cost matrix to be optimized, further eliminating accumulated errors and occlusion artifacts that may occur during dynamic programming. The two methods work together: the former ensures the overall smoothness and edge preservation of the disparity field, while the latter ensures the accuracy and geometric consistency of local details. Together, they solve the problems of poor anti-interference capability in traditional methods and the loss of detail and weak generalization in pure end-to-end deep learning methods. The final output aggregated cost matrix has both high signal-to-noise ratio and high geometric accuracy, supporting the subsequent generation of high-quality disparity maps with sub-pixel accuracy.

[0063] In one or more embodiments of this example, the step of performing pixel disparity processing on the first view and the second view based on the aggregation cost matrix to obtain a target disparity map includes: According to the preset screening strategy, a target aggregation cost is selected in the aggregation cost matrix, and the initial integer disparity corresponding to the target aggregation cost is determined; continuous domain fitting is performed on the cost value in the neighborhood of the initial integer disparity, and the target disparity map is determined based on the fitting result.

[0064] Specifically, the pre-defined screening strategy can refer to the rule of selecting the optimal matching result from multiple candidate disparities in the aggregated cost matrix based on the magnitude of their cost values. Specifically, this strategy can employ the Winner-Takes-All (WTA) algorithm, which, for each pixel in the image, iterates through its corresponding one-dimensional cost vector in the aggregated cost matrix and uses the disparity with the minimum cost value as the best matching estimate for that pixel. The initial integer-pixel disparity is an integer-level depth estimate obtained based on the discrete search space, derived from the process of searching for global or local minima in the aggregated cost matrix. The initial integer-pixel disparity... This step compresses the high-dimensional aggregation cost data into a single-channel preliminary depth map, providing baseline coordinates for subsequent refinement. For example, if a pixel in the aggregation cost matrix has a cost of 0.8, 0.2, and 0.9 at disparities of 12, 13, and 14, respectively, a preset filtering strategy will determine that disparity 13 is the initial integer-pixel disparity corresponding to the target aggregation cost. This filtering mechanism quickly identifies the most likely matching position, effectively suppressing false matches caused by noise interference and laying a reliable integer foundation for subsequent sub-pixel refinement.

[0065] Correspondingly, continuous domain fitting can refer to the process of modeling the cost distribution near discrete disparity points using mathematical functions to estimate the precise minimum points at non-integer locations. The initial integer-pixel disparity neighborhood typically includes the initial disparity and its left and right adjacent integer disparity points (i.e.,...). , , Specifically, the fitting process often employs parabolic fitting, using the disparity values ​​of the three discrete points as the abscissa and the corresponding aggregated cost value as the ordinate to construct a quadratic function curve. The abscissa corresponding to the minimum point of this curve is then calculated, thereby obtaining the sub-pixel level target disparity. This step is used to overcome the resolution limitations of discrete sampling in digital images, improving disparity accuracy from integer to fractional levels. For example, if the initial integer pixel disparity is 13, the cost values ​​of its left and right neighboring disparities 12 and 14 are relatively high. Through parabolic fitting, the minimum point may be calculated to be located at 13.45, and this 13.45 is the finally determined sub-pixel disparity.

[0066] In practical implementation, to compensate for the lack of fine disparity refinement in deep learning classification and regression, traditional parabolic interpolation can be used to achieve sub-pixel level, extremely high-precision disparity refinement. This utilizes the minimum cost point... The cost of the disparity to its left and right neighbors is used to calculate the final subpixel disparity d. sub The calculation formula is as follows: Formula (3): (3) Finally, you can choose to introduce a traditional left-right consistency check to eliminate erroneous parallax in occluded areas.

[0067] Based on this, after obtaining the aggregation cost matrix, in order to ensure that the subsequent view construction is more accurate, the target aggregation cost can be selected in the aggregation cost matrix according to the preset filtering strategy. Then, the initial integer disparity corresponding to the target aggregation cost can be determined. Based on this, continuous domain fitting can be performed on the cost value in the neighborhood of the initial integer disparity, so as to determine the target disparity map according to the fitting result.

[0068] In summary, high-precision stereo matching was achieved through the above processing. Specifically, firstly, a preset screening strategy was used to quickly locate the globally optimal integer disparity candidates in the aggregated cost matrix, ensuring global consistency in matching. Based on this, continuous domain fitting was further performed on the cost value within the initial integer pixel disparity neighborhood, using the geometric characteristics of the local cost function to derive sub-pixel-level precise positions. This combination of discrete coarse selection and continuous refinement inherits the rigor of traditional semi-global matching algorithms in terms of geometric constraints while compensating for the quantization errors introduced by discretization search. Through continuous domain processing techniques such as parabolic fitting, the system can resolve depth changes smaller than one pixel unit, thus maintaining robustness in weakly textured regions while outputting highly faithful depth information at object edges and fine structures, significantly improving the detail restoration capability and measurement accuracy of the final target disparity map.

[0069] In one or more embodiments of this example, after the step of performing pixel disparity processing on the first view and the second view based on the aggregation cost matrix to obtain the target disparity map is executed, the method further includes: The target disparity map is optimized, and a target depth map is generated based on the optimized target disparity map; the motion trajectory of the target robot in the current scene is constructed based on the target depth map, and the target robot is driven to move based on the motion trajectory.

[0070] Specifically, optimizing the target disparity map can refer to the process of removing mismatched points and filling in empty areas. This optimization process includes performing a left-right consistency check, mapping the disparity calculated from the left view to the right view, and then back to the left view. If the difference between the two calculated disparity values ​​exceeds a preset threshold, the pixel is identified as an occluded or mismatched area and marked for removal. Subsequently, a hole-filling algorithm is used to interpolate and repair the removed area using the disparity values ​​of surrounding valid pixels, thus obtaining the optimized target disparity map. Generating a target depth map from the optimized target disparity map can refer to using the camera's imaging geometry model to convert the optimized disparity values ​​into physical distance information. Disparity and depth are inversely proportional; that is, the depth value equals the product of the baseline distance and the focal length divided by the disparity value. For example, when the baseline distance of a binocular camera is 0.12 meters and the focal length is 800 pixels, if the optimized disparity value of a pixel is 40 pixels, then the calculated depth value for that point is 2.4 meters.

[0071] Correspondingly, constructing a motion trajectory based on the target depth map can refer to using Simultaneous Localization and Mapping (SLAM) technology or path planning algorithms to reconstruct a 3D point cloud map or occupancy grid map from a 2D target depth map, and then identifying passable areas and obstacles in this map. Specifically, the system first calculates the 3D coordinates of the obstacle in the robot coordinate system based on the depth value of each pixel in the depth map and the camera intrinsic parameters; then, it uses path search strategies such as the A* algorithm, Dijkstra's algorithm, or artificial potential field method to plan a collision-free and cost-minimizing motion trajectory between the current pose and the target pose. Driving the target robot to move based on the motion trajectory can refer to the robot's motion control system discretizing the planned trajectory into a series of speed and steering commands, and controlling the wheel assembly or foot actuators through motor drivers. For example, when the depth map detects an obstacle with a height of 0.5 meters 1.5 meters ahead, the path planning module automatically generates a detour trajectory, and the control system then outputs a command to turn 30 degrees to the left and decelerate forward, driving the robot to smoothly bypass the obstacle. It achieves a closed loop from visual perception to action control. Through the precise spatial information provided by the depth map, the target robot can autonomously complete obstacle avoidance, navigation and fixed-point movement tasks in complex dynamic environments, which significantly improves the robot's operation capability and safety in unstructured scenarios.

[0072] In summary, the above processing achieves a complete technical chain from image perception to autonomous robot motion. First, by optimizing the target disparity map, matching noise and occlusion artifacts are effectively removed, ensuring the generated target depth map has extremely high geometric accuracy. Then, a motion trajectory is constructed based on the high-precision target depth map, enabling the path planning algorithm to make decisions based on real 3D environmental information, avoiding collision risks or path failures caused by depth errors. Finally, the robot moves based on this reliable trajectory, not only leveraging the robustness of the hybrid stereo matching method in weak texture and highly reflective scenes, but also directly converting the computational results of the visual algorithm into actual control signals for the robot. This cascaded architecture of depth feature extraction - traditional cost aggregation - disparity optimization - depth reconstruction - trajectory planning retains the strong generalization ability of deep learning in feature representation while utilizing the advantages of traditional geometric methods in sub-pixel accuracy and spatial consistency, thus significantly improving the target robot's perception accuracy, decision reliability, and motion control smoothness in complex scenes.

[0073] See Figure 2 , Figure 2 A flowchart of another image processing method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0074] Step S202: Obtain multiple views and input the multiple views into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module.

[0075] Step S204: Use the feature extraction module to extract the view features corresponding to each view.

[0076] Step S206: Using the matrix construction module, perform matching cost calculation for the view features corresponding to each view within a preset disparity search range, and generate an initial cost matrix based on the calculation results.

[0077] Step S208: For the multiple matching costs contained in the initial cost matrix, cost aggregation is performed to obtain an aggregated cost matrix, and pixel disparity processing is performed on the multiple views based on the aggregated cost matrix to obtain a target disparity map.

[0078] Another image processing method provided in this embodiment is applied to the implementation of stereo matching based on two or more views. For any content not described herein, please refer to the same or corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0079] Corresponding to the above method embodiments, this specification also provides embodiments of image processing apparatus. Figure 3A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification is shown. Figure 3 As shown, the device includes: The acquisition module 302 is configured to acquire a first view and a second view, and input the first view and the second view into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The extraction module 304 is configured to extract first view features corresponding to the first view and second view features corresponding to the second view using the feature extraction module. The calculation module 306 is configured to use the matrix construction module to perform matching cost calculation on the first view feature and the second view feature within a preset disparity search range, and generate an initial cost matrix based on the calculation result; The processing module 308 is configured to perform cost aggregation on multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and to perform pixel disparity processing on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

[0080] In an optional embodiment, obtaining the first view and the second view, and inputting the first view and the second view into the image processing model, includes: A first view is acquired through a first image acquisition device of the target robot, and a second view is acquired through a second image acquisition device of the target robot; the first view and the second view are input into an image processing model, wherein the image processing model uses a weight-shared neural network as a feature extraction module, and the feature extraction module is used to extract view features containing multi-scale receptive fields and high-dimensional semantics.

[0081] In an optional embodiment, the step of using the matrix construction module to calculate the matching cost for the first view feature and the second view feature within a preset disparity search range, and generating an initial cost matrix based on the calculation result, includes: Using the matrix construction module, the matching cost corresponding to each preset disparity within the preset disparity search range is calculated for the first view feature and the second view feature; and an initial cost matrix is ​​constructed based on the matching cost corresponding to each preset disparity.

[0082] In an optional embodiment, determining the matching cost corresponding to any preset disparity includes: The matching cost corresponding to the preset disparity is obtained by calculating the inner product of the first view feature and the second view feature under a preset disparity using the matrix construction module; or, the matching cost corresponding to the preset disparity is obtained by calculating the distance between the first view feature and the second view feature under a preset disparity using the matrix construction module.

[0083] In an optional embodiment, the step of aggregating the costs of multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix includes: The initial cost matrix contains multiple matching costs, which are aggregated under path constraints in multiple spatial directions using a semi-global matching algorithm to obtain an aggregated cost matrix to be optimized. The aggregated cost matrix to be optimized is then optimized according to a preset cost optimization strategy to obtain an aggregated cost matrix.

[0084] In an optional embodiment, the step of performing pixel disparity processing on the first view and the second view based on the aggregation cost matrix to obtain a target disparity map includes: According to the preset screening strategy, a target aggregation cost is selected in the aggregation cost matrix, and the initial integer disparity corresponding to the target aggregation cost is determined; continuous domain fitting is performed on the cost value in the neighborhood of the initial integer disparity, and the target disparity map is determined based on the fitting result.

[0085] In an optional embodiment, after the step of performing pixel disparity processing on the first view and the second view based on the aggregation cost matrix to obtain the target disparity map, the method further includes: The target disparity map is optimized, and a target depth map is generated based on the optimized target disparity map; the motion trajectory of the target robot in the current scene is constructed based on the target depth map, and the target robot is driven to move based on the motion trajectory.

[0086] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0087] Corresponding to the above method embodiments, this specification also provides another embodiment of an image processing apparatus. Figure 4 A schematic diagram of another image processing apparatus provided in one embodiment of this specification is shown. Figure 4 As shown, the device includes: The view acquisition module 402 is configured to acquire multiple views and input the multiple views into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module 404 is configured to extract view features corresponding to each view using the feature extraction module; The matrix generation module 406 is configured to use the matrix construction module to perform matching cost calculation within a preset disparity search range for the view features corresponding to each view, and generate an initial cost matrix based on the calculation results. The cost aggregation module 408 is configured to perform cost aggregation on multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and to perform pixel disparity processing on the multiple views based on the aggregated cost matrix to obtain a target disparity map.

[0088] The above is an illustrative scheme of another image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0089] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0090] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0091] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0092] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0093] The processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0094] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image processing method described above.

[0095] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0096] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image processing method described above.

[0097] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0098] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image processing method described above.

[0099] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0100] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0101] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0102] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0103] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. An image processing method, characterized in that, include: A first view and a second view are obtained, and the first view and the second view are input into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module is used to extract the first view features corresponding to the first view and the second view features corresponding to the second view. The matrix construction module is used to calculate the matching cost for the first view feature and the second view feature within a preset disparity search range, and an initial cost matrix is ​​generated based on the calculation results. Cost aggregation is performed on the multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix. Then, pixel disparity processing is performed on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

2. The image processing method according to claim 1, characterized in that, The step of obtaining a first view and a second view, and inputting the first view and the second view into an image processing model, includes: A first view is acquired using a first image acquisition device of the target robot, and a second view is acquired using a second image acquisition device of the target robot; The first view and the second view are input into an image processing model, wherein the image processing model uses a weight-shared neural network as a feature extraction module, and the feature extraction module is used to extract view features that include multi-scale receptive fields and high-dimensional semantics.

3. The image processing method according to claim 1, characterized in that, The step of using the matrix construction module to calculate the matching cost for the first view feature and the second view feature within a preset disparity search range, and generating an initial cost matrix based on the calculation result, includes: Using the matrix construction module, the matching cost corresponding to each preset disparity within the preset disparity search range is calculated for the first view feature and the second view feature; An initial cost matrix is ​​constructed based on the matching cost corresponding to each preset disparity.

4. The image processing method according to claim 3, characterized in that, Determining the matching cost corresponding to any preset disparity includes: The matrix construction module calculates the inner product of the first view feature and the second view feature under a preset disparity to obtain the matching cost corresponding to the preset disparity; or... The matrix construction module is used to calculate the distance between the first view feature and the second view feature under a preset disparity, and the matching cost corresponding to the preset disparity is obtained.

5. The image processing method according to claim 1, characterized in that, The step of aggregating the costs of multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix includes: The initial cost matrix contains multiple matching costs, which are aggregated under path constraints in multiple spatial directions using a semi-global matching algorithm to obtain the aggregated cost matrix to be optimized. The aggregate cost matrix to be optimized is optimized according to a preset cost optimization strategy to obtain the aggregate cost matrix.

6. The image processing method according to claim 1, characterized in that, The step of performing pixel disparity processing on the first view and the second view based on the aggregation cost matrix to obtain a target disparity map includes: According to a preset filtering strategy, a target aggregation cost is selected from the aggregation cost matrix, and the initial integer disparity corresponding to the target aggregation cost is determined. A continuous domain fitting is performed on the cost value within the initial integer pixel disparity region, and the target disparity map is determined based on the fitting result.

7. The image processing method according to claim 2, characterized in that, After the step of performing pixel disparity processing on the first view and the second view based on the aggregation cost matrix to obtain the target disparity map is executed, the method further includes: The target disparity map is optimized, and a target depth map is generated based on the optimized target disparity map; Based on the target depth map, the motion trajectory of the target robot in the current scene is constructed, and the target robot is driven to move based on the motion trajectory.

8. An image processing method, characterized in that, include: Multiple views are acquired and input into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module is used to extract the view features corresponding to each view. Using the matrix construction module, matching cost is calculated for the view features corresponding to each view within a preset disparity search range, and an initial cost matrix is ​​generated based on the calculation results. Cost aggregation is performed on the multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix. Then, pixel disparity processing is performed on the multiple views based on the aggregated cost matrix to obtain a target disparity map.

9. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire a first view and a second view, and input the first view and the second view into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The extraction module is configured to extract first view features corresponding to the first view and second view features corresponding to the second view using the feature extraction module. The calculation module is configured to use the matrix construction module to perform matching cost calculation on the first view feature and the second view feature within a preset disparity search range, and generate an initial cost matrix based on the calculation result; The processing module is configured to perform cost aggregation on multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and to perform pixel disparity processing on the first view and the second view based on the aggregated cost matrix to obtain a target disparity map.

10. An image processing apparatus, characterized in that, include: The view acquisition module is configured to acquire multiple views and input the multiple views into an image processing model, wherein the image processing model includes a feature extraction module and a matrix construction module; The feature extraction module is configured to extract view features corresponding to each view using the feature extraction module; The matrix generation module is configured to use the matrix construction module to perform matching cost calculation within a preset disparity search range for the view features corresponding to each view, and generate an initial cost matrix based on the calculation results. The cost aggregation module is configured to aggregate costs for multiple matching costs contained in the initial cost matrix to obtain an aggregated cost matrix, and to perform pixel disparity processing on the multiple views based on the aggregated cost matrix to obtain a target disparity map.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images