Humanoid robot multi-camera feature fusion mode

Abstract

A humanoid robot multi-camera feature fusion method, comprising a plurality of cameras; it includes the following steps: calibrating each camera to obtain the intrinsic and extrinsic parameters of the corresponding camera; according to the intrinsic parameters of each camera, the horizontal field of view angle of the corresponding camera is calculated; according to the horizontal field of view angle and the extrinsic parameters of each camera, the coverage range of the plurality of cameras on the horizontal reference surface centered on the robot is obtained; a plurality of cameras respectively collect images to obtain a fusion projection image; the fusion projection image is input into an image recognition model to obtain the position of the object relative to the robot; the above technical scheme effectively avoids the problem that the neural network model is too large and the training and reasoning cost is high due to the direct use of high-dimensional image data, significantly reduces the consumption of computing resources, and also improves the accuracy and efficiency of detection, quickly and accurately locating the target object in the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-camera system fusion, and in particular to a humanoid robot multi-camera feature fusion method. BACKGROUND

[0002] The multi-camera system fusion technology on the market is constantly developing, aiming to improve the accuracy and robustness of target recognition, tracking and scene understanding. With the continuous progress of technology and the continuous expansion of application scenarios, multi-camera fusion technology will be applied in more fields, such as urban security monitoring, autonomous driving, etc. The existing multi-camera system fusion technology mainly includes decision layer fusion and feature layer fusion.

[0003] Decision layer fusion refers to independent data processing of information collected by each camera, such as image preprocessing, target detection and recognition, etc. A deep learning model (such as YOLOv3, YOLOv5 or a more advanced 3D target detection network) is usually used to predict the 3D position, category and confidence of the target. The process of fusing the recognition results is to unify the obtained information into the robot coordinate system. This level of fusion is carried out at the decision layer, and its result often provides the basis for command and control decision-making; however, decision layer fusion often requires setting up independent neural networks for each camera, which is not conducive to maintaining time synchronization during deployment. At the same time, the neural networks corresponding to the number of cameras will occupy a serious GPU performance, and make the entire autonomous driving system very complex, prone to system errors.

[0004] Feature layer fusion refers to first extracting high-level features from the original image using a convolutional neural network (CNN) for the information collected by each camera, such as the edges, shapes, textures and motion information of the target, etc. Then, through methods such as BEVFusion, the features of different cameras are mapped to the same space by a network (such as Transformer) to realize cross-fusion of features. Feature fusion often needs to use transformer or cross-attention mechanism as the basic structure of the fusion network. These mechanisms usually contain a large number of matrix multiplication and self-attention calculation, resulting in high computational complexity. This is a great challenge for resource-constrained board robots, for example, the adaptability of NPU to transformer is very poor, and often needs to write an operator separately for transformer processing on CPU during deployment. SUMMARY

[0005] In view of the above problems, the present application provides a humanoid robot multi-camera feature fusion method, which solves the problems of high computational complexity and large resource consumption of existing feature layer fusion.

[0006] To achieve the above object, the inventor provides a human-shaped robot multi-camera feature fusion method, comprising several cameras; it comprises the following steps:

[0007] Calibrate each camera to obtain the intrinsic and extrinsic parameters of the corresponding camera;

[0008] According to the intrinsic parameters of each camera, the horizontal field of view angle of the corresponding camera is calculated;

[0009] According to the horizontal field of view angle and the extrinsic parameters of each camera, the coverage range of several cameras on the horizontal reference plane centered on the robot is obtained;

[0010] Several cameras capture images to obtain a fusion projection image;

[0011] The fusion projection image is input into an image recognition model to obtain the position of the object relative to the robot.

[0012] Further, the direction of the lens of each camera points to a direction parallel to the horizontal reference plane.

[0013] Further, in the step of calculating the horizontal field of view angle of the corresponding camera according to the intrinsic parameters of each camera,

[0014] Set the focal length (f x , f y ), principal point coordinates (c x , c y ) in the intrinsic parameters of the camera;

[0015] Then the horizontal field of view angle HFOV of the camera is Arc tan((image_w / 2-c x ) / f x ), wherein image_w is the width of the camera image.

[0016] Further, in the step of several cameras capturing images to obtain a fusion projection image, the resolution of the fusion projection image is also set.

[0017] Further, the step of inputting the fusion projection image into the image recognition model to obtain the position of the target object relative to the robot, comprising:

[0018] Input the fusion projection image into the image recognition model to obtain the center coordinates and yaw angle of the bounding box in the fusion projection image;

[0019] According to the center coordinates and yaw angle of the bounding box and the resolution of the fusion projection image, the position of the target object relative to the vehicle body is obtained.

[0020] Further, in the step of obtaining the position of the target object relative to the robot in the inputting the fusion projection image into the image recognition model, the image recognition model is a pre-trained deep learning network model.

[0021] Further, the deep learning network model is a YOLOv9 model or a model improved based on the YOLOv9 model.

[0022] Further, the model improved based on the YOLOv9 model includes adding an auxiliary branch to two.

[0023] Further, the model improved based on the YOLOv9 model includes replacing the standard convolution block in the GELAN module with a reparameterizable convolution.

[0024] Further, the model improved based on the YOLOv9 model includes using anchor free as the detection head and adding a target object yaw angle regression layer based on anchor free.

[0025] Unlike the prior art, the above technical solution mainly uses the imaging geometric principle to accurately project the image data captured by multiple cameras from different positions and different angles onto a unified horizontal reference plane centered on the robot, constructing a fusion projection image with multiple perspectives of information. The fusion projection image comprehensively reflects the target object layout and spatial relationship in the monitoring scene from different perspectives. Then, a deep learning network model is introduced to perform in-depth feature extraction on the fusion projection image, extracting key features closely related to the target object, such as shape, edge, etc., and obtaining accurate position information of the target object relative to the robot center. This effectively avoids the problem of a too large neural network model, high training and inference cost caused by directly using high-dimensional image data, significantly reduces the consumption of computing resources, and improves the accuracy and efficiency of detection, quickly and accurately locating the target object in the image.

[0026] The above invention content is only a summary of the technical solution of the present application. In order to enable those skilled in the art to more clearly understand the technical solution of the present application, and then implement it according to the content of the description and the drawings, and in order to make the above and other purposes, features and advantages of the present application more easily understood, the following will be described in conjunction with the specific embodiments of the present application and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present application and other related content, and cannot be considered as a limitation of the present application.

[0028] In the drawings of the specification:

[0029] Figure 1 A flowchart illustrating the multi-camera feature fusion method for the humanoid robot described in a specific implementation;

[0030] Figure 2 This is a schematic diagram illustrating the coverage area of ​​the plurality of cameras on a horizontal reference plane centered on the robot, as described in a specific embodiment.

[0031] Figure 3 This is a schematic diagram of the image recognition model structure described in a specific implementation. Detailed Implementation

[0032] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0033] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0034] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0035] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0036] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0037] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0038] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0039] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0040] See Figures 1-3 As shown, a multi-camera feature fusion method for humanoid robots mainly utilizes imaging geometry principles to accurately project image data captured by multiple cameras from different positions and angles onto a uniformly defined horizontal reference plane centered on the robot. This constructs a fused projection image with information from multiple perspectives, comprehensively reflecting the layout and spatial relationships of target objects in the monitoring scene from different viewpoints. Subsequently, a deep learning network model is introduced to perform in-depth feature extraction on the fused projection image, extracting key features closely related to the target object, such as shape and edges, and obtaining precise positional information of the target object relative to the robot's center. This effectively avoids the problems of excessively large neural network models and high training and inference costs caused by directly using high-dimensional image data, significantly reducing the consumption of computing resources, while also improving the accuracy and efficiency of detection, and quickly and accurately locating target objects in the image.

[0041] SeeFigure 1 As shown, the following describes a multi-camera feature fusion method for a humanoid robot, which includes several cameras; the main steps are as follows:

[0042] S1. Calibrate each camera to obtain the corresponding camera's intrinsic and extrinsic parameters;

[0043] S2. Calculate the horizontal field of view of each camera based on its intrinsic parameters;

[0044] S3. Based on the horizontal field of view and extrinsic parameters of each camera, obtain the coverage area of ​​several cameras on the horizontal reference plane centered on the robot.

[0045] S4. Several cameras capture images separately to obtain a fused projection image;

[0046] S5. Input the fused projection image into the image recognition model to obtain the position of the object relative to the robot.

[0047] The lenses of the aforementioned cameras can be pointed parallel or nearly parallel to the horizontal reference plane, ensuring that their field of view primarily covers the horizontal direction of the scene, rather than the vertical direction. Preferably, the lens of each camera is parallel to the horizontal reference plane, eliminating roll and pitch angles. This minimizes image projection errors caused by lens tilt (deviations in the position, shape, and size of target objects in the image from reality). It accurately projects image data captured by cameras from different positions and angles onto a uniformly defined horizontal reference plane centered on the robot, forming an accurate fused projection image. This provides more reliable and accurate image data support for subsequent image recognition, target tracking, and intelligent analysis functions.

[0048] In step S1, the intrinsic and extrinsic parameters of each camera can be obtained through calibration of the corresponding camera. The calibration process may involve photographing a calibration object with a known geometric shape (such as a checkerboard pattern or a dot array), placing the calibration object at different positions and orientations within the camera's field of view to obtain sufficient calibration data, and then extracting the edges and corners of the calibration object using an image processing algorithm. Based on the extracted corner coordinates and the known dimensions of the calibration object, the intrinsic and extrinsic parameters of the camera are calculated using a calibration algorithm (such as the Zhang Zhengyou calibration method). The intrinsic parameters of the camera include parameters such as focal length and principal point coordinates, describing the internal geometric characteristics of the camera; the extrinsic parameters of the camera include rotation and translation matrices, describing the camera's position and orientation in the world coordinate system.

[0049] In step S2 above, the step of calculating the horizontal field of view of the corresponding camera based on the intrinsic parameters of each camera...

[0050] Set the focal length (f) in the camera's intrinsic parameters. x f y ), principal point coordinates (c x c y );

[0051] Then the horizontal field of view HFOV of the camera is equal to Arctan((image_w / 2-c). x ) / f x The image resolution (image_h, image_w) is defined as follows: image_w represents the width of the camera image, which is the number of pixels (usually in pixels) in the horizontal direction. For example, the resolution of an image captured by a camera (image_h, image_w) is defined as follows: image_w represents the number of pixels in the horizontal direction, while image_h represents the number of pixels in the vertical direction.

[0052] See Figure 2 As shown, step S3 above, which obtains the coverage area of ​​several cameras on a horizontal reference plane centered on the robot based on the horizontal field of view and extrinsic parameters of each camera, includes:

[0053] Based on the horizontal field of view and extrinsic parameters of the camera, the coverage area of ​​a single camera on a horizontal reference plane centered on the robot is calculated using trigonometric functions and geometric principles.

[0054] Overlapping areas are detected by comparing the coverage areas projected by adjacent cameras.

[0055] Within the overlapping area, the coverage area of ​​one camera is selected as the effective area, and the coverage area of ​​the other camera is removed, so as to form the coverage area of ​​several cameras on a horizontal reference plane centered on the robot.

[0056] Specifically, an image is captured once using several cameras. Based on the horizontal field of view and extrinsic parameters of the cameras, the image pixel coordinates (x, y) are projected onto a horizontal reference plane (X, Y) centered on the robot, as shown in the following formula:

[0057] Y = Z / Arctan((yc y ) / f y )

[0058] X = Y / Arctan((xc x ) / f x )

[0059] Where Z is the vertical distance from the camera to the ground;

[0060] The location and extent of overlapping regions are precisely determined by using image matching algorithms or feature point detection algorithms. Within the overlapping region, the image from one camera is selected as the valid area, and portions of the image from the other camera are discarded, thereby determining the coverage area of ​​several cameras on a horizontal reference plane centered on the robot.

[0061] In step S4 above, several cameras acquire images to obtain a fused projection image. This is mainly achieved by projecting the images onto a horizontal reference plane centered on the robot, based on the coverage area of ​​the existing cameras on the horizontal reference plane. The projected images are then stitched and fused to form the fused projection image. The resolution of the fused projection image can be a default setting or can be set separately to calculate the position of the target object relative to the robot based on the resolution of the fused projection image.

[0062] Step S5 above, which involves inputting the fused projection image into the image recognition model to obtain the position of the object relative to the robot, includes:

[0063] The fused projection image is input into the image recognition model to obtain the center coordinates and yaw angle of the bounding box in the fused projection image;

[0064] The position of the object relative to the vehicle body is obtained based on the center coordinates and yaw angle of the bounding box and the resolution of the fused projection image.

[0065] Specifically, the center coordinates of the bounding box are scaled according to the resolution of the fused projection image. The yaw angle θ can be directly used to represent the rotation direction of the target object relative to the vehicle body.

[0066] The aforementioned image recognition model should be able to detect target objects in an image and output the center coordinates and yaw angle of the bounding box. This image recognition model is a pre-trained deep learning network model. Through training, the deep learning network model can learn the correlations and patterns between these features, thereby achieving effective recognition and analysis of target objects, enabling the image recognition model to accurately obtain the center coordinates and yaw angle of the bounding box in the fused projection image. The deep learning network model can be trained using a projection image dataset. Preferably, the projection image dataset includes projection images of the same target object in different scenes on a horizontal reference plane, and / or projection images of different target objects in the same scene on a horizontal reference plane, to improve the robustness of the image recognition model. The specific training process is as follows: the projection images in the projection image dataset are segmented and labeled, and the results are used as labels for the training data. The image recognition model is trained using the projection images in the projection image dataset. The output of the image recognition model is calculated through forward propagation, and then the difference between the output and the labels of the training data is calculated using a loss function. The parameters of the image recognition model are updated through a backpropagation algorithm to obtain the final image recognition model. In some embodiments, before segmenting, labeling, and using the projected images in the projected image dataset as training data, the projected images in the projected image dataset can be preprocessed, such as through denoising or data augmentation. For example, data augmentation can involve rotating, scaling, or flipping the projected images in the projected image dataset to improve the generalization ability and adaptability of the image recognition model.

[0067] The aforementioned deep learning network model can be based on a Convolutional Neural Network (CNN), the YOLOE series, or other deep learning models, or even improved models based on the above models, and is trained on a large number of projection images to enable the image recognition model to accurately obtain the center coordinates and yaw angle of the bounding box in the fused projection image. To further illustrate the image recognition model of this invention, the YOLOE series model is used as an example to further explain the deep learning network model. The deep learning network model can be the YOLOv9 model. The image recognition model is obtained by training the YOLOv9 model to accurately obtain the center coordinates and yaw angle of the bounding box in the fused projection image. Considering that the recognized fused projection image is an approximate surround-view projection image, and considering the deployment capability on different robots, the YOLOv9 model is partially modified, i.e., an improved model based on the YOLOv9 model is used. See [link to relevant documentation]. Figure 3 As shown, specific improvements can include the following:

[0068] The improved YOLOv9 model includes adding two auxiliary branches. Specifically, two auxiliary branches are added after the backbone of the YOLOv9 network, utilizing one more auxiliary branch to train the parameters of the backbone. The backbone is responsible for the final prediction task, while the auxiliary branches provide more gradient information to the backbone during training, helping it learn features better. Increasing the number of auxiliary branches to two ensures that more gradient information flows to the backbone during training, thus helping the backbone learn richer and more robust features (such as...). Figure 3 As shown in the diagram, the dual auxiliary branches simply copy the part within the red box once.

[0069] The improved YOLOv9 model replaces the standard convolutional blocks in the GELAN module with reparameterizable convolutions. By replacing the standard convolutional blocks with reparameterizable convolutions, more complex convolutional operations can be used to learn stronger feature representations during the training phase, while these complex convolutional operations can be simplified to standard convolutional operations during the inference phase, thus maintaining the model's lightweight nature and fast inference speed.

[0070] The improved YOLOv9 model uses an anchor-free detection head and adds a yaw angle regression layer on top of it. The anchor-free mechanism avoids the anchor box design and matching problems of anchor-based methods, allowing the model to more flexibly adapt to target objects of different scales and shapes, improving generalization ability, reducing hyperparameter setting and tuning, and thus improving detection accuracy. The yaw angle regression layer predicts the yaw angle of the target object, obtaining more accurate object orientation information, enabling the image recognition model to better handle object detection tasks in complex scenes. Furthermore, as a component of the detection head, the yaw angle regression layer shares the feature extraction network with the original detection task, requiring no additional computational resources.

[0071] The processor described in the embodiments of this application can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0072] The computer program involved in the embodiments can be stored in a computer robot-readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc. It also includes other biological, physical, or chemical structures capable of performing similar or equivalent functions to the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types or a combination of the above media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium or distributed across multiple media. The memory containing the computer robot-readable storage medium can be non-volatile memory or random access memory. These computer robot-readable storage media can be built into the robot or connected to the robot involved in the embodiments as an external robot or part of an external robot. In some embodiments, the memory with a computer robot-readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as it enables computer robot access to the memory. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0073] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A multi-camera feature fusion method for a humanoid robot, comprising a plurality of cameras; characterized in that, Includes the following steps: Each camera is calibrated to obtain its intrinsic and extrinsic parameters. Calculate the horizontal field of view of the corresponding camera based on the intrinsic parameters of each camera; in the step of calculating the horizontal field of view of the corresponding camera based on the intrinsic parameters of each camera, Set the focal length in the camera's intrinsic parameters. Principal point coordinates ; The horizontal field of view of the camera Among them The width of the camera image; Based on the horizontal field of view and extrinsic parameters of each camera, the coverage area of ​​several cameras on a horizontal reference plane centered on the robot is obtained; Several cameras capture images separately to obtain a fused projection image; The fused projection image is input into the image recognition model to obtain the position of the object relative to the robot; The step of inputting the fused projection image into the image recognition model to obtain the position of the target object relative to the robot includes: The fused projection image is input into the image recognition model to obtain the center coordinates of the bounding box in the fused projection image. horn; Based on the center coordinates of the bounding box and By analyzing the angle and the resolution of the fused projection image, the position of the target object relative to the vehicle body is obtained.

2. The multi-camera feature fusion method for humanoid robots according to claim 1, characterized in that, The lens of each camera is pointing in a direction parallel to the horizontal reference plane.

3. The multi-camera feature fusion method for humanoid robots according to claim 1, characterized in that, The step of acquiring images from several cameras to obtain a fused projection image also includes setting the resolution of the fused projection image.

4. The multi-camera feature fusion method for humanoid robots according to claim 1, characterized in that, In the step of obtaining the position of the target object relative to the robot by inputting the fused projection image into the image recognition model, the image recognition model is a pre-trained deep learning network model.

5. The multi-camera feature fusion method for humanoid robots according to claim 4, characterized in that, The deep learning network model is the YOLOv9 model or a model improved based on the YOLOv9 model.

6. The multi-camera feature fusion method for humanoid robots according to claim 5, characterized in that, The improved model based on the YOLOv9 model includes adding auxiliary branches to two lines.

7. The multi-camera feature fusion method for humanoid robots according to claim 5, characterized in that, The improved model based on the YOLOv9 model includes replacing the standard convolutional blocks in the GELAN module with reparameterizable convolutions.

8. The multi-camera feature fusion method for humanoid robots according to claim 5, characterized in that, The improved YOLOv9 model includes using anchor-free as the detection head and adding a target object on top of the anchor-free model. The regression layer of the angle.

Images