A target point position measurement method, system and storage medium

Abstract

The application provides a target point position measurement method, system and computer readable storage medium. The method comprises the following steps: acquiring multiple camera images via multiple cameras; analyzing a first camera image via a first network module of a pre-trained semantic point measurement network model to determine first plane coordinate information and first semantic information of at least one first object; analyzing a second camera image via a second network module of the semantic point measurement network model to determine second plane coordinate information and second semantic information of at least one second object; and via a third network module of the semantic point measurement network model, performing graph neural network-based matching on the first plane coordinate information, the first semantic information, the second plane coordinate information and the second semantic information according to a relative spatial relationship between the multiple cameras to determine at least one target point of at least one object with the same semantic information and to determine a spatial coordinate position of the target point.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more particularly to a method for measuring the position of a target point, a system for measuring the position of a target point, and a computer-readable storage medium. Background Technology

[0002] Binocular cameras are commonly used vision sensors in fields such as autonomous driving and robotics. They can simultaneously capture images of a target using two relatively fixed lenses, and then calculate the three-dimensional position of the target point in the image relative to the camera based on the different content of the captured images and the relative spatial relationship between the two cameras (camera parameters).

[0003] Automatic recognition refers to the process of finding the position (pixel coordinates) of a target point in an image. The challenge lies in the fact that due to differences in lighting, target texture, and shooting position and pose, the target point appears differently in the image, making direct recognition impossible using preset conditions such as grayscale values ​​or shape. Automatic measurement refers to simultaneously obtaining the pixel values ​​of a target point in two images and calculating the target point's three-dimensional position relative to the camera by combining camera parameters. The challenge lies in the fact that due to the different perspectives of the two cameras, the same target point appears differently in the images from the two cameras, and occlusion can even prevent the target point from being seen in one image, ultimately leading to a mismatch between the target points in the two camera images. This problem is also known as the binocular matching problem. Therefore, both the automatic recognition problem and the automatic binocular matching problem need to be solved simultaneously to ultimately achieve automatic measurement.

[0004] In existing technologies, solutions to the problem of automatic target point recognition are generally divided into two categories: traditional methods and deep learning methods. Traditional methods, represented by keypoint algorithms such as Harris corner detection and Scale Invariant Feature Transform (SIFT), can only define keypoints based on explicit conditions such as edges and corners, and record the set of values ​​calculated from image grayscale (such as grayscale gradients) as the feature description vector of the keypoint. However, they cannot calculate high-level semantic information such as whether the keypoint is a vertex of a license plate quadrilateral or the endpoint of a utility pole, and therefore cannot directly obtain the target point. Deep learning methods mainly include target point recognition algorithms, which can obtain high-level semantic information such as the endpoint of a utility pole, but cannot calculate the feature description vector of the point, and therefore cannot be used for stereo matching.

[0005] Furthermore, solutions to the binocular matching problem can be categorized into two types: traditional methods and deep learning methods. Traditional methods, represented by brute-force matching and grid motion matching, primarily rely on the feature description vector of each point for matching calculations. Deep learning methods, represented by OANet (Order-Aware Network) and SuperGlue algorithms, use deep learning networks to solve the matching problem function, but they also depend on the feature description vector of each point. Therefore, a common problem with these algorithms is that the feature description and binocular matching processes are computed separately by two independent models. One model must first identify the target point and its feature description, and then another model must be used for target point matching, resulting in a slower overall computation process and making it unsuitable for scenarios with high real-time requirements.

[0006] In order to overcome the above-mentioned defects of the existing technology, there is an urgent need in the field for a target point location measurement technology that can synchronously parse and obtain the planar coordinate information and semantic information of the target object through a semantic point measurement network model, and perform target point matching based on graph neural network to improve the real-time performance and accuracy of target point recognition, and make the model easier to train. Summary of the Invention

[0007] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0008] To overcome the aforementioned deficiencies in the existing technology, the present invention provides a method for measuring the location of a target point, a system for measuring the location of a target point, and a computer-readable storage medium. This system can synchronously parse and obtain the planar coordinate information and semantic information of the target object through a semantic point measurement network model, and perform target point matching based on a graph neural network to improve the real-time performance and accuracy of target point recognition, and make the model easier to train.

[0009] Specifically, the target point location measurement method provided by the first aspect of the present invention includes the following steps: acquiring multiple camera images via multiple cameras; parsing the first camera images via a first network module of a pre-trained semantic point measurement network model to determine first planar coordinate information and first semantic information of at least one first object; parsing the second camera images via a second network module of the semantic point measurement network model to determine second planar coordinate information and second semantic information of at least one second object; and performing graph neural network-based matching of the first planar coordinate information, the first semantic information, the second planar coordinate information, and the second semantic information based on the relative spatial relationship between the multiple cameras via a third network module of the semantic point measurement network model to determine at least one target point of at least one object having the same semantic information, and determining the spatial coordinate location of the target point.

[0010] Preferably, in one embodiment of the present invention, the first network module and / or the second network module respectively include a detection network unit and an intermediate computing unit. The step of parsing the camera image to determine the planar coordinate information and semantic information of at least one object includes: performing a first convolution processing on the acquired camera image via the detection network unit to obtain multiple feature maps of different dimensions; and inputting each of the feature maps into multiple input ports of corresponding dimensions of the intermediate computing unit, and performing a second convolution processing on each of the feature maps via the intermediate computing unit to obtain planar coordinate information and semantic information of multiple corresponding dimensions.

[0011] Preferably, in one embodiment of the present invention, the detection network unit includes a slicing layer, and the step of performing a first convolutional process on the acquired camera image via the detection network unit to obtain multiple feature maps of different dimensions includes: slicing the acquired camera image into RGB layers via the slicing layer to obtain grayscale images of multiple color channels; and performing a first convolutional process on each of the grayscale images to obtain multiple feature maps of different dimensions for each of the color channels.

[0012] Preferably, in one embodiment of the present invention, the step of performing a first convolutional process on the acquired camera image via the detection network unit to obtain multiple feature maps of different dimensions includes: processing the acquired camera image via a first convolutional layer and a first cross-stage residual block to obtain a first feature map of a first dimension; and processing the first feature map via a second convolutional layer, a second cross-stage residual block, and a pooling bottleneck layer to obtain a second feature map of a second dimension, wherein the second dimension is less than the first dimension.

[0013] Preferably, in one embodiment of the present invention, the step of performing a second convolutional process on each of the feature maps via the intermediate computing unit to obtain planar coordinate information and semantic information of multiple corresponding dimensions includes: processing the second feature map via a third convolutional layer to obtain a first intermediate vector; upsampling the first intermediate vector to obtain a second intermediate vector; combining the first feature map and the second intermediate vector, and processing the combined vector via a third cross-stage residual block to obtain a third intermediate vector; processing the third intermediate vector via a first output head to obtain planar coordinate information and semantic information of the first dimension; downsampling the third intermediate vector to obtain a fourth intermediate vector; combining the first intermediate vector and the fourth intermediate vector, and processing the combined vector via a fourth cross-stage residual block to obtain a fifth intermediate vector; and processing the fifth intermediate vector via a second output head to obtain planar coordinate information and semantic information of the second dimension.

[0014] Preferably, in one embodiment of the present invention, the output head includes a detection box output port and a key point detection output port. The step of processing the intermediate vector through the output head to obtain the planar coordinate information and the semantic information includes: processing the intermediate vector through the detection box output port to obtain a detection box output [x1, y1, w1, h1, class1], wherein x1 and y1 indicate the vertex coordinates of a specified corner of the detection box, w1 indicates the width of the detection box, h1 indicates the height of the detection box, and class1 records the semantic information corresponding to the detection box; and processing the intermediate vector through the key point detection output port to obtain a key point output [x2, y2, class2], wherein x2 and y2 indicate the coordinates of the key point, and class2 records the semantic information corresponding to the key point.

[0015] Preferably, in one embodiment of the present invention, the key point detection output port is configured with a descriptor decoding unit, the descriptor decoding unit including at least one VGG subunit, an interpolation subunit and a normalization subunit, the step of processing the intermediate vector through the key point detection output port to obtain the key point output [x2, y2, class2] includes: processing the intermediate vector sequentially through the at least one VGG subunit, the interpolation subunit and the normalization subunit to obtain a semantic feature map describing the semantic information of the key point.

[0016] Preferably, in one embodiment of the present invention, the third network module includes a matching network unit and a binocular computing unit. The step of the third network module, via the semantic point measurement network model, performing graph neural network-based matching on the first planar coordinate information, the first semantic information, the second planar coordinate information, and the second semantic information according to the relative spatial relationship between the plurality of cameras to determine at least one target point of at least one object having the same semantic information, and determining the spatial coordinate position of the target point includes: performing graph neural network-based matching on the first semantic feature map of the first camera image and the second camera image via the matching network unit to determine at least one target point of at least one object having the same semantic information; and determining the spatial coordinate position of the target point via the binocular computing unit according to the relative spatial relationship between the plurality of cameras, the first planar coordinate information of the target point in the first camera image, and the second planar coordinate information of the target point in the second camera image.

[0017] Furthermore, the target point location measurement system provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the target point location measurement method provided in any of the above embodiments.

[0018] Furthermore, the computer-readable storage medium provided according to the third aspect of the present invention stores computer instructions. When the computer instructions are executed by a processor, the method for measuring the position of a target point provided in any of the above embodiments is implemented. Attached Figure Description

[0019] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0020] Figure 1 A schematic diagram of the architecture of a target point location measurement system provided according to some embodiments of the present invention is shown.

[0021] Figure 2 A flowchart of a method for measuring the location of a target point according to some embodiments of the present invention is shown.

[0022] Figure 3 A schematic diagram of the architecture of a detection network unit provided according to some embodiments of the present invention is shown.

[0023] Figure 4A schematic diagram of the architecture of an intermediate computing unit provided according to some embodiments of the present invention is shown.

[0024] Figure 5 A schematic diagram of the architecture of an output head provided according to some embodiments of the present invention is shown.

[0025] Figure 6 A schematic diagram of the architecture of the detection key point output port provided according to some embodiments of the present invention is shown.

[0026] Figure 7 The diagram shows the matching effect of human hair endpoints according to some embodiments of the present invention. Detailed Implementation

[0027] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0028] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0029] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0030] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0031] As mentioned above, existing solutions to the problem of automatic target point recognition are generally divided into two categories: traditional methods and deep learning methods. Traditional methods, represented by keypoint algorithms such as Harris corner detection and Scale Invariant Feature Transform (SIFT), can only define keypoints based on explicit conditions such as edges and corners, and record the set of values ​​calculated from image grayscale (such as grayscale gradients) as the feature description vector of the keypoint. However, they cannot calculate high-level semantic information such as whether the keypoint is a vertex of a license plate quadrilateral or an endpoint of a utility pole, and therefore cannot directly obtain the target point. Deep learning methods mainly include target point recognition algorithms, which can obtain high-level semantic information such as the endpoint of a utility pole, but cannot calculate the feature description vector of the point, and therefore cannot be used for stereo matching.

[0032] Furthermore, solutions to the binocular matching problem can be categorized into two types: traditional methods and deep learning methods. Traditional methods, represented by brute-force matching and grid motion matching, primarily rely on the feature description vector of each point for matching calculations. Deep learning methods, represented by OANet (Order-Aware Network) and the SuperGlu algorithm, use deep learning networks to solve the matching problem function, but they also depend on the feature description vector of each point. Therefore, a common problem with these algorithms is that the feature description and binocular matching processes are computed separately by two independent models. One model must first identify the target point and its feature description, and then another model must be used for target point matching, resulting in a slower overall computation process and making it unsuitable for scenarios with high real-time requirements.

[0033] To overcome the aforementioned deficiencies in the existing technology, the present invention provides a method for measuring the location of a target point, a system for measuring the location of a target point, and a computer-readable storage medium. This system can synchronously parse and obtain the planar coordinate information and semantic information of the target object through a semantic point measurement network model, and perform target point matching based on a graph neural network to improve the real-time performance and accuracy of target point recognition, and make the model easier to train.

[0034] In some non-limiting embodiments, the target point location measurement method provided in the first aspect of the present invention can be implemented via the target point location measurement system provided in the second aspect of the present invention. Specifically, the target point location measurement system may be configured with a memory and a processor. The memory includes, but is not limited to, the computer-readable storage medium provided in the third aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the target point location measurement method provided in the first aspect of the present invention.

[0035] The working principle of the target point location measurement system described below will be described with reference to embodiments of some target point location measurement methods. Those skilled in the art will understand that these embodiments of measurement methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or operating methods of the measurement system. Similarly, this measurement system is also only one non-limiting implementation provided by the present invention and does not limit the executing entities or execution order of the steps in these measurement methods.

[0036] Please refer to the reference. Figure 1 and Figure 2 . Figure 1 A schematic diagram of the architecture of a target point location measurement system provided according to some embodiments of the present invention is shown. Figure 2 A flowchart of a method for measuring the location of a target point according to some embodiments of the present invention is shown.

[0037] In such Figure 1 In the illustrated embodiment, the target point location measurement system provided by the second aspect of the present invention can be structured on a Semantic Point Measure Net (SPMNet) model, including a first network module and a second network module respectively connected to the left eye camera 21 and the right eye camera 22, and a third network module located at the back end of the first network module and the second network module.

[0038] Furthermore, the first network module includes a detection network unit 111 and an intermediate computing unit 112, used to parse the first camera image output by the left eye camera 21 to determine the first planar coordinate information and first semantic information of at least one first object.

[0039] The second network module includes a detection network unit 121 and an intermediate computing unit 122, used to parse the second camera image output by the right eye camera 22 to determine the second planar coordinate information and second semantic information of at least one second object.

[0040] The third network module includes a matching network unit 131 and a binocular computing unit 132, which is used to perform graph neural network-based matching on the first planar coordinate information, the first semantic information, the second planar coordinate information and the second semantic information according to the relative spatial relationship between the multiple cameras, so as to determine at least one target point of at least one object with the same semantic information, and determine the spatial coordinate position of the target point.

[0041] like Figure 2 As shown, in the process of measuring the position of the target point, the target point position measurement system can first acquire multiple camera images through multiple cameras 21-22, and then parse the first camera images through the first network module of the pre-trained SPMNet model to determine the first planar coordinate information and first semantic information of at least one first object. Then, it can parse the second camera images through the second network module of the SPMNet model to determine the second planar coordinate information and second semantic information of at least one second object.

[0042] Specifically, in the process of parsing camera images to determine the planar coordinate information and semantic information of at least one object, the measurement system can first perform a first convolution process on the acquired camera images through each detection network unit 111 and 121 to obtain multiple feature maps of different dimensions. Then, each feature map is input into the input ports of the corresponding intermediate computing units 112 and 122 of multiple corresponding dimensions. Each intermediate computing unit 112 and 122 performs a second convolution process on the corresponding feature map to obtain planar coordinate information and semantic information of multiple corresponding dimensions.

[0043] Please refer to the reference. Figures 3-5 . Figure 3 A schematic diagram of the architecture of a detection network unit provided according to some embodiments of the present invention is shown. Figure 4 A schematic diagram of the architecture of an intermediate computing unit provided according to some embodiments of the present invention is shown. Figure 5 A schematic diagram of the architecture of an output head provided according to some embodiments of the present invention is shown.

[0044] like Figure 3 As shown, in some embodiments of the present invention, the detection network units 111 and 121 may include a slicing layer 301 for RGB slicing the input image. In response to acquiring camera images from cameras 21-22, the detection network units 111 and 121 may first perform RGB slicing on the acquired camera images via the slicing layer 301 to obtain grayscale images of multiple color channels such as RGB, and then input each grayscale image into the convolutional block, cross-stage residual block and spatial pyramid pooling bottleneck layer below, and perform a first convolutional processing on each grayscale image to obtain multiple feature maps of different dimensions for each color channel.

[0045] Specifically, in response to the grayscale image output by the slice layer 301, the detection network units 111 and 121 can first perform convolution processing on the image via the convolutional layer 302 and the cross-stage residual block 303 to obtain a preliminary feature map. Then, the detection network units 111 and 121 can perform a first convolution processing on the preliminary feature map via the convolutional layer 311 and the cross-stage residual block 312 to obtain a first feature map with a first dimension. Furthermore, the detection network units 111 and 121 can also perform a second convolution processing on the first feature map via the convolutional layer 321 and the cross-stage residual block 322 to obtain a second feature map with a second dimension, perform a third convolution processing on the second feature map via the convolutional layer 331, the cross-stage residual block 332, and the pooling bottleneck layer 333 to obtain a third feature map with a third dimension, and perform a fourth convolution processing on the third feature map via the convolutional layer 341, the cross-stage residual block 342, and the pooling bottleneck layer 343 to obtain a fourth feature map with a fourth dimension. Here, the number of the first to fourth dimensions can be different to improve the measurement accuracy of the SPMNet model. For example, the first, second, third, and fourth convolution processes can all be downsampling processes to reduce the number of the first dimension (e.g., 600). 600) is greater than the number of the second dimension (e.g., 400). 400), making the second dimension (e.g., 400) 400) is greater than the third dimension (e.g., 300). 300), and make the third dimension (e.g., 300) 300) is greater than the fourth dimension (e.g., 200). 200).

[0046] Furthermore, such as Figure 4As shown, intermediate computing units 112 and 122 can have a neural network architecture with four dimensions, corresponding to detection network units 111 and 121. In response to acquiring feature maps of four different dimensions from the output interfaces 1-4 of detection network units 111 and 121, intermediate computing units 112 and 122 can first process the fourth feature map via convolutional layer 441 to obtain a first intermediate vector, and then upsample the first intermediate vector via upsampling unit 442 to obtain a second intermediate vector. Next, intermediate computing units 112 and 122 can combine the acquired third feature map and the second intermediate vector via combination unit 431 to obtain a third intermediate vector, and then process the third intermediate vector via cross-stage residual block 432 to obtain a fourth intermediate vector. Following this, intermediate computing units 112 and 122 can perform convolution processing on the acquired second feature map and the third intermediate vector via convolutional layer 421 to obtain a fifth intermediate vector, and then upsample the fifth intermediate vector via upsampling unit 422 to obtain a sixth intermediate vector. Subsequently, intermediate computing units 112 and 122 can combine the acquired first feature map and the sixth intermediate vector through the combination unit 411, process the combined vector obtained through the cross-stage residual block 412 to obtain the seventh intermediate vector, and then process the seventh intermediate vector through the first output head 51 to obtain the planar coordinate information and semantic information of the first dimension.

[0047] Furthermore, intermediate computing units 112 and 122 can continue to downsample the aforementioned seventh intermediate vector via downsampling unit 413 to obtain an eighth intermediate vector, and then combine the aforementioned fifth and eighth intermediate vectors via combination unit 423 to obtain a ninth intermediate vector. Afterwards, intermediate computing units 112 and 122 can process the ninth intermediate vector via cross-stage residual block 424 to obtain a tenth intermediate vector, and then process the tenth intermediate vector via second output head 52 to obtain planar coordinate information and semantic information of the second dimension.

[0048] Furthermore, intermediate computing units 112 and 122 can further downsample the aforementioned ninth intermediate vector via downsampling unit 425 to obtain an eleventh intermediate vector, and then combine the aforementioned fourth intermediate vector and the eleventh intermediate vector via combination unit 433 to obtain a twelfth intermediate vector. Afterwards, intermediate computing units 112 and 122 can process the twelfth intermediate vector via cross-stage residual block 434 to obtain a thirteenth intermediate vector, and then process the thirteenth intermediate vector via third output head 53 to obtain planar coordinate information and semantic information of the third dimension.

[0049] Furthermore, intermediate computing units 112 and 122 can further downsample the aforementioned twelfth intermediate vector via downsampling unit 435 to obtain the fourteenth intermediate vector, and then combine the aforementioned first intermediate vector and the fourteenth intermediate vector via combination unit 443 to obtain the fifteenth intermediate vector. Afterwards, intermediate computing units 112 and 122 can process the fifteenth intermediate vector via cross-stage residual block 444 to obtain the sixteenth intermediate vector, and then process the sixteenth intermediate vector via fourth output head 54 to obtain the planar coordinate information and semantic information of the fourth dimension.

[0050] Furthermore, such as Figure 5 As shown, the output heads 51-54 may include detection box output ports 511-541 and detection keypoint output ports 512-542, respectively. In response to the intermediate vectors output by the intermediate computing units 112 and 122, the output heads 51-54 can process the intermediate vectors of corresponding dimensions via the detection box output ports 511-541 to obtain the corresponding detection box outputs [x1, y1, w1, h1, class1]. Here, x1 and y1 indicate the vertex coordinates of a specified corner of the detection box, w1 indicates the width of the detection box, h1 indicates the height of the detection box, and class1 records high-level semantic information indicating the object category, such as "cat" or "dog". Furthermore, in response to the intermediate vectors output by the intermediate computing units 112 and 122, the output heads 51-54 can also process the intermediate vectors of corresponding dimensions via the detection keypoint output ports 512-542 to obtain the corresponding keypoint outputs [x2, y2, class2]. Here, x2 and y2 indicate the coordinates of the key point, and class2 records the high-level semantic information corresponding to the key point, such as neighborhood information, gray value, and gradient value.

[0051] Please refer to further information. Figure 6 , Figure 6 A schematic diagram of the architecture of the detection key point output port provided according to some embodiments of the present invention is shown.

[0052] exist Figure 6 In the illustrated embodiment, the keypoint detection output ports 512-542 may preferably be configured with a descriptor decoding unit, specifically including at least one VGG subunit, an interpolation subunit, and a normalization subunit. In response to the intermediate vector output by the intermediate calculation units 112 and 122, the keypoint detection output ports 512-542 may first... The intermediate vector of dimension is input into a 3D array. The 3VGG subunit 61 is convolved, and its output 256-dimensional variable is then input into a 1 The VGG subunit 62 of unit 1 undergoes convolution processing, and then a custom adjustment descriptor channel number D is introduced and input together with it into the BiCubic interpolation subunit 63 for interpolation processing to obtain h. w The semantic feature vector is obtained in D dimensions. Then, the keypoint detection output ports 512-542 can input this semantic feature vector into the normalization subunit 64 for normalization processing to obtain the corresponding dimension (h). w D) The semantic feature map describing the semantic information of key points is used to adapt to the input requirements of the matching network unit 131 in the subsequent third network module. Here, the semantic feature maps of each output interface 1 to 4 can have different values ​​of h, w, and D depending on the first, second, third, and fourth dimensions mentioned above, which will not be elaborated here.

[0053] Those skilled in the art will understand that Figures 3-5 The neural network architecture shown is for extracting planar coordinate information and semantic information based on four different dimensions. It is only a non-limiting implementation of the present invention and is intended to clearly demonstrate the main concept of the present invention and provide a preferred solution to improve the measurement accuracy of the target point position, rather than to limit the scope of protection of the present invention.

[0054] Alternatively, in other embodiments, those skilled in the art can also construct semantic point measurement network models with two, three, or more layers of neural network architecture based on the above concept to achieve the same function of target point position measurement and obtain the corresponding measurement accuracy.

[0055] For example, in a semantic point measurement network model containing only a two-layer neural network architecture, the detection network unit can first process the grayscale image output from the slice layer through a first convolutional layer and a first cross-stage residual block to obtain a first feature map of a first dimension. Then, it processes this first feature map through a second convolutional layer, a second cross-stage residual block, and a pooling bottleneck layer to obtain a second feature map of a second dimension. Further, this first dimension (e.g., 600) (600) can be greater than the second dimension (e.g., 200). (200), thereby improving the measurement accuracy of the target point position by extracting planar coordinate information and semantic information of different dimensions.

[0056] Correspondingly, the intermediate computation unit can be a neural network architecture containing only two layers of different dimensions, similar to the detection network unit. In response to obtaining two feature maps of different dimensions from the output interfaces of the detection network unit, the intermediate computation unit can first process the second feature map via a third convolutional layer to obtain a first intermediate vector, and then upsample the first intermediate vector to obtain a second intermediate vector. Subsequently, the intermediate computation unit can combine the first feature map and the second intermediate vector, and process the combined vector via a third cross-stage residual block to obtain a third intermediate vector. This third intermediate vector is then processed by the first output head to obtain planar coordinate information and semantic information of the first dimension.

[0057] Furthermore, the intermediate computing unit can continue to downsample the aforementioned third intermediate vector to obtain a fourth intermediate vector, and combine the aforementioned first intermediate vector and the fourth intermediate vector. This combined vector is then processed by the fourth cross-stage residual block to obtain a fifth intermediate vector. Afterward, the intermediate computing unit can process the fifth intermediate vector via the second output head to obtain planar coordinate information and semantic information of the second dimension.

[0058] Please continue to refer to this. Figure 1 and Figure 2 In response to the first planar coordinate information and first semantic information output by the first network module, and the second planar coordinate information and second semantic information output by the second network module, the third network module can further obtain the relative spatial relationship between each camera 21, 22, and perform graph neural network-based matching on the first planar coordinate information, first semantic information, second planar coordinate information, and second semantic information to determine at least one target point of at least one object with the same semantic information, and determine the spatial coordinate position of the target point. Here, the relative spatial relationship between each camera 21, 22 may optionally include camera intrinsic parameters such as focal length and depth of field, and camera extrinsic parameters such as relative position and translation matrix.

[0059] Specifically, the third network module can first run the matching network unit 131 via the graphics processing unit (GPU) to perform graph neural network-based matching on the first semantic feature map of the first camera image and the second camera image to determine at least one target point of at least one object with the same semantic information. Then, via the binocular computing unit 132, the spatial coordinates of the target point are determined based on the relative spatial relationship between the multiple cameras 21 and 22, the first planar coordinate information of the target point in the first camera image, and the second planar coordinate information of the target point in the second camera image. Here, the matching function can be implemented using open-source algorithms such as SuperGlue, and the function of determining the spatial coordinates of the target point can also be implemented using existing spatial conversion methods. Neither of these involves technical improvements of the present invention and will not be elaborated here.

[0060] Thus, by adapting the output feature map to the matching network unit 131 based on the graph neural network, the present invention can realize automatic forward propagation calculation from extracting feature description vectors to target matching, thereby avoiding additional computational overhead between modules to the greatest extent, significantly improving the computational speed of the entire process, and being suitable for scenarios with high real-time requirements. On the other hand, it can make it easier to control the final accuracy index during network training.

[0061] Furthermore, compared to existing technologies that use a Central Processing Unit (CPU) for target point matching, this invention significantly improves the computational speed of target point matching by employing a GPU to perform graph neural network-based matching on feature maps representing the first plane coordinate information and the first semantic information, as well as the second plane coordinate information and the second semantic information output by the second network module. For example, when using a GPU RTX3090 to match 1024... In applications involving target point measurement using 1024 resolution camera images, a measurement speed of 50ms per pair of images is achieved.

[0062] Please refer to further information. Figure 7 , Figure 7 The diagram shows the matching effect of human hair endpoints according to some embodiments of the present invention.

[0063] exist Figure 7 In the illustrated embodiment, the objects connected by lines represent the same hair strand that has been matched. Due to their high density and texture similarity, they are difficult to identify and match. By employing the SPMNet model based on deep learning to extract feature description vectors of key points, this invention achieves higher accuracy and greater robustness compared to traditional feature matching methods.

[0064] In summary, the target point location measurement method, target point location measurement system, and computer-readable storage medium provided by this invention only require multiple camera images acquired by multiple cameras and the relative spatial relationships between the multiple cameras to automatically measure the spatial coordinate position of the target point relative to the cameras, thus exhibiting high real-time performance. Furthermore, by synchronously parsing and acquiring the planar coordinate information and semantic information of the target object via a semantic point measurement network model, and performing target point matching based on a graph neural network, this invention can directly obtain matching target points with the same semantic information from the image input, thereby effectively improving the real-time performance and accuracy of target point recognition and making the model easier to train.

[0065] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0066] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and skills. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0067] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0068] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for measuring the position of a target point, characterized in that, Includes the following steps: Multiple camera images are acquired using multiple cameras; The first camera image is parsed by the first network module of the pre-trained semantic point measurement network model to determine the first planar coordinate information and first semantic information of at least one first object therein; The second camera image is parsed via the second network module of the semantic point measurement network model to determine the second planar coordinate information and second semantic information of at least one second object therein; as well as The third network module of the semantic point measurement network model performs graph neural network-based matching on the first planar coordinate information, the first semantic information, the second planar coordinate information, and the second semantic information according to the relative spatial relationship between the multiple cameras, in order to determine at least one target point of at least one object with the same semantic information, and determine the spatial coordinate position of the target point.

2. The measurement method as described in claim 1, characterized in that, The first network module and / or the second network module respectively include a detection network unit and an intermediate computing unit. The step of parsing the camera image to determine the planar coordinate information and semantic information of at least one object includes: The detection network unit performs a first convolution process on the acquired camera images to obtain multiple feature maps of different dimensions; and Each of the aforementioned feature maps is input into the input ports of the intermediate computing unit at multiple corresponding dimensions. The intermediate computing unit then performs a second convolution process on each of the feature maps to obtain planar coordinate information and semantic information at multiple corresponding dimensions.

3. The measurement method as described in claim 2, characterized in that, The detection network unit includes a slicing layer. The step of performing a first convolutional process on the acquired camera image via the detection network unit to obtain multiple feature maps of different dimensions includes: The acquired camera image is sliced ​​in RGB using the slicing layer to obtain a grayscale image with multiple color channels; and Each grayscale image is subjected to a first convolution process to obtain multiple feature maps of different dimensions for each color channel.

4. The measurement method as described in claim 2, characterized in that, The step of performing a first convolutional process on the acquired camera image via the detection network unit to obtain multiple feature maps of different dimensions includes: Camera images obtained through processing by the first convolutional layer and the first cross-stage residual block are used to obtain a first feature map of a first dimension; and The first feature map is processed by a second convolutional layer, a second cross-stage residual block, and a pooling bottleneck layer to obtain a second feature map with a second dimension, wherein the second dimension is less than the first dimension.

5. The measurement method as described in claim 4, characterized in that, The step of performing a second convolution process on each feature map via the intermediate computing unit to obtain planar coordinate information and semantic information of multiple corresponding dimensions includes: The second feature map is processed via a third convolutional layer to obtain a first intermediate vector; Upsample the first intermediate vector to obtain the second intermediate vector; The first feature map and the second intermediate vector are combined, and their combined vector is processed by a third cross-stage residual block to obtain a third intermediate vector; The third intermediate vector is processed via the first output head to obtain the planar coordinate information and semantic information of the first dimension. The third intermediate vector is downsampled to obtain the fourth intermediate vector; The first intermediate vector and the fourth intermediate vector are combined, and their combined vector is processed via a fourth cross-stage residual block to obtain a fifth intermediate vector; and The fifth intermediate vector is processed via the second output head to obtain the planar coordinate information and semantic information of the second dimension.

6. The measurement method as described in claim 5, characterized in that, The output head includes a detection box output port and a detection key point output port. The step of processing the intermediate vector through the output head to obtain the planar coordinate information and the semantic information includes: The intermediate vector is processed via the detection box output port to obtain the detection box output [x1, y1, w1, h1, class1], where x1 and y1 indicate the vertex coordinates of a specified corner of the detection box, w1 indicates the width of the detection box, h1 indicates the height of the detection box, and class1 records the semantic information corresponding to the detection box; and The intermediate vector is processed through the key point detection output port to obtain key point output [x2, y2, class2], where x2 and y2 indicate the coordinates of the key point, and class2 records the semantic information corresponding to the key point.

7. The measurement method as described in claim 6, characterized in that, The keypoint detection output port is equipped with a descriptor decoding unit, which includes at least one VGG subunit, an interpolation subunit, and a normalization subunit. The step of processing the intermediate vector through the keypoint detection output port to obtain the keypoint output [x2, y2, class2] includes: The intermediate vector is processed sequentially by the at least one VGG subunit, the interpolation subunit, and the normalization subunit to obtain a semantic feature map describing the semantic information of the key point.

8. The measurement method as described in claim 7, characterized in that, The third network module includes a matching network unit and a binocular computing unit. The third network module, via the semantic point measurement network model, performs graph neural network-based matching on the first planar coordinate information, the first semantic information, the second planar coordinate information, and the second semantic information according to the relative spatial relationship between the multiple cameras to determine at least one target point of at least one object with the same semantic information, and determines the spatial coordinate position of the target point. The matching network unit performs graph neural network-based matching on the first semantic feature map of the first camera image and the second camera image to determine at least one target point of at least one object having the same semantic information; and The spatial coordinates of the target point are determined by the binocular computing unit based on the relative spatial relationship between the multiple cameras, the first planar coordinate information of the target point in the first camera image, and the second planar coordinate information of the target point in the second camera image.

9. A system for measuring the position of a target point, characterized in that, include: Memory, on which computer instructions are stored; as well as A processor, connected to the memory, and configured to execute computer instructions stored in the memory to implement the method for measuring the position of a target point as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the method for measuring the position of the target point as described in any one of claims 1 to 8 is implemented.

Images