Vehicle-mounted mechanical arm control method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of vehicle-mounted mechanical arm control, and provides a vehicle-mounted mechanical arm control method, device and equipment, a storage medium and a program product. The method comprises the steps that in-vehicle image data is obtained through real-time shooting of a camera set; inputting the in-vehicle image data into the feature extraction model in a time frame alignment synchronization mode to obtain a position information graph; the position information graph comprises position information of each entity; the feature extraction model is used for performing fusion extraction on the image features of the at least two cameras through the mixed attention network module, and positioning the moving position of the vehicle-mounted mechanical arm in combination with the coordinate change matrix; and according to the position information graph and the voice instruction obtained through the microphone, the vehicle-mounted mechanical arm is controlled to execute an operation task corresponding to the voice instruction. According to the invention, a camera group is adopted to carry out image multi-view shooting, and then combined modeling is carried out to carry out image feature fusion extraction; and meanwhile, the voice recognition technology is integrated, and auxiliary application of the vehicle-mounted intelligent mechanical arm is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle-mounted mechanical arm control, and in particular to a vehicle-mounted mechanical arm control method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of smart car technology, consumers hope that smart cars can provide more convenient functions. In the current technical solutions, there is no mature application of intelligent robotic arms in the car. Most of the intelligent robotic arms in general fields rely on monocular cameras. However, this kind of intelligent robotic arm based on monocular cameras has significant limitations, which are mainly reflected in the following two aspects: Single-view problem: Monocular cameras can only obtain image information from a fixed viewpoint, making it difficult to identify objects or scenes in obscured areas. In complex environments, this single-view limitation can lead to inaccurate recognition.

[0003] Limited perception range: Monocular cameras have limited perception distance and width, and cannot effectively identify areas outside the field of view. The contradiction between viewing angle and detection range further limits its recognition ability in complex environments.

[0004] These problems show that intelligent robotic arms that rely solely on monocular cameras can hardly meet the needs of smart cars for efficient and accurate recognition. Summary of the invention

[0005] The present invention provides a vehicle-mounted robotic arm control method, device, equipment, storage medium and program product to solve the problem in the prior art that smart cars lack auxiliary applications of vehicle-mounted intelligent robotic arms with high efficiency and accurate identification.

[0006] The present invention provides a vehicle-mounted mechanical arm control method, which is arranged inside a vehicle. The vehicle-mounted mechanical arm control method comprises: acquiring in-vehicle image data through real-time shooting of a camera group; wherein the camera group comprises at least two cameras; the in-vehicle image data comprises a plurality of in-vehicle entities; the in-vehicle image data is input into a feature extraction model in a time frame alignment synchronization manner to obtain a position information map output by the feature extraction model; wherein the position information map comprises position information of each entity; the feature extraction model is used to fuse and extract image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted mechanical arm in combination with a coordinate change matrix; and according to the position information map and the voice command obtained through a microphone, the vehicle-mounted mechanical arm is controlled to execute an operation task corresponding to the voice command.

[0007] According to a vehicle-mounted robotic arm control method provided by the present invention, at least two cameras include a first camera and a second camera; the in-vehicle image data include first in-vehicle image data obtained by the first camera and second in-vehicle image data obtained by the second camera; the in-vehicle image data are input into a feature extraction model in a time frame alignment and synchronization manner, including: determining the synchronization deviation of two frames of data in the first in-vehicle image data and the second in-vehicle image data according to a hardware synchronization signal between the first camera and the second camera; if the synchronization deviation is less than or equal to a preset deviation threshold, inputting the first in-vehicle image data and the second in-vehicle image data into the feature extraction model for feature extraction; if the synchronization deviation is greater than the preset deviation threshold, obtaining each frame signal of the first camera and the second camera for similarity matching, and when the similarity value is greater than the preset similarity threshold, inputting the first in-vehicle image data and the second in-vehicle image data into the feature extraction model for feature extraction; when the similarity value is less than or equal to the preset similarity threshold, determining that the synchronization fails, re-collecting new first in-vehicle image data through the first camera, and re-collecting new second in-vehicle image data through the second camera for time frame alignment and synchronization.

[0008] According to a vehicle-mounted robotic arm control method provided by the present invention, a feature extraction model performs feature extraction, channel dimension reduction and data preprocessing on first in-vehicle image data and second in-vehicle image data respectively to obtain a first feature data block and a second feature data block of the same specification; the first feature data block and the second feature data block are subjected to data fusion through a hybrid attention network module to obtain a fused feature block; a position information map is determined based on the fused feature block, and the coordinates of the vehicle-mounted robotic arm are determined through the position information map and the coordinate change matrix to locate the moving position of the vehicle-mounted robotic arm.

[0009] According to a vehicle-mounted robotic arm control method provided by the present invention, before the in-vehicle image data is input into the feature extraction model in a time frame alignment and synchronization manner, it also includes: calibrating the camera group and the vehicle-mounted robotic arm to establish a motion coordinate system; and determining the coordinate change matrix according to the motion coordinate system.

[0010] According to a vehicle-mounted robotic arm control method provided by the present invention, the vehicle-mounted robotic arm is controlled to perform an operation task corresponding to the voice command based on a position information map and a voice command obtained through a microphone, including: obtaining a voice command through a microphone; parsing the voice command to obtain a target object and a target position; the target object is determined from multiple entities in the vehicle according to the voice command; according to the target object, the target position and the position information map, a motion path for the vehicle-mounted robotic arm to move the target object to the target position is planned; according to the motion path, the vehicle-mounted robotic arm is controlled to perform the operation task of moving the target object to the target position.

[0011] According to a vehicle-mounted robotic arm control method provided by the present invention, multiple in-vehicle entities include a human body; the position information map includes human body posture information; the human body posture information includes hand posture information and / or face posture information; the vehicle-mounted robotic arm control method also includes: according to the hand posture information and the motion path, controlling the vehicle-mounted robotic arm to move the target object from the hand to the target position; or, controlling the vehicle-mounted robotic arm to move the target object from the original position to the hand as the target position; and / or, according to the face posture information and voice commands, obtaining the target object and target position.

[0012] The present invention also provides a vehicle-mounted robotic arm control device, which is arranged inside the vehicle, and the vehicle-mounted robotic arm control device includes: an in-vehicle image data acquisition module, which is used to acquire in-vehicle image data through real-time shooting of a camera group; wherein the camera group includes at least two cameras; the in-vehicle image data includes multiple in-vehicle entities; a feature extraction model module, which is used to input the in-vehicle image data into the feature extraction model in a time frame alignment synchronization manner, and obtain a position information map output by the feature extraction model; wherein the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted robotic arm in combination with the coordinate change matrix; a robotic arm control module, which is used to control the vehicle-mounted robotic arm to perform operation tasks corresponding to the voice commands according to the position information map and the voice commands obtained through the microphone.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described vehicle-mounted robotic arm control methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the vehicle-mounted mechanical arm control method described in any one of the above methods is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the vehicle-mounted mechanical arm control method described in any one of the above methods is implemented.

[0016] The vehicle-mounted mechanical arm control method, device, equipment, storage medium and program product provided by the present invention, the vehicle-mounted mechanical arm is arranged inside the vehicle, and the vehicle-mounted mechanical arm control method includes: obtaining in-vehicle image data through real-time shooting of a camera group; wherein the camera group includes at least two cameras; the in-vehicle image data includes multiple in-vehicle entities; the in-vehicle image data is input into the feature extraction model in a time frame alignment synchronization manner to obtain a position information map output by the feature extraction model; wherein the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted mechanical arm in combination with the coordinate change matrix; according to the position information map and the voice command obtained through the microphone, the vehicle-mounted mechanical arm is controlled to perform the operation task corresponding to the voice command. In the above manner, the present invention can fully identify the behavior activities of the driver and / or passengers in the vehicle and provide them with high-quality auxiliary services. The present invention adopts a camera group to shoot images from multiple perspectives, and then performs joint modeling to perform image feature fusion extraction. It further proposes a method for cascading a robotic arm and a camera, and integrates voice recognition technology to realize an auxiliary application device for an on-board intelligent robotic arm, which can provide a convenient usage scenario for the future smart car living space. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a vehicle-mounted mechanical arm control method provided in an embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of the connection of in-vehicle hardware devices provided by an embodiment of the present invention.

[0020] Figure 3 It is a schematic diagram of the layout position of the vehicle-mounted mechanical arm provided in an embodiment of the present invention.

[0021] Figure 4 It is a schematic diagram of the structure of a vehicle-mounted mechanical arm provided in an embodiment of the present invention.

[0022] Figure 5 It is a schematic diagram of a vehicle-mounted mechanical arm grabbing an object provided by an embodiment of the present invention.

[0023] Figure 6 It is a schematic diagram of the folding of the vehicle-mounted mechanical arm provided by an embodiment of the present invention.

[0024] Figure 7 It is a schematic diagram of a network architecture for feature extraction based on multiple cameras provided in an embodiment of the present invention.

[0025] Figure 8 It is a structural schematic diagram of a vehicle-mounted mechanical arm control device provided in an embodiment of the present invention.

[0026] Fig. 9 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limitations on the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0029] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0030] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0031] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0032] The present invention provides a vehicle-mounted mechanical arm control method, please refer to Figure 1 , Figure 1 : is a flow chart of a vehicle-mounted mechanical arm control method provided by an embodiment of the present invention. In this embodiment, the vehicle-mounted mechanical arm control method may include steps S110 to S130, each of which is specifically as follows: S110: Acquire in-vehicle image data through real-time shooting of the camera group.

[0033] The camera group includes at least two cameras; and the in-vehicle image data includes multiple in-vehicle entities.

[0034] S120: Inputting the in-vehicle image data into the feature extraction model in a time frame alignment and synchronization manner to obtain a position information map output by the feature extraction model.

[0035] Among them, the location information map includes the location information of each entity; the feature extraction model is used to fuse and extract the image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted robotic arm in combination with the coordinate change matrix.

[0036] S130: According to the position information map and the voice command obtained through the microphone, the vehicle-mounted robotic arm is controlled to execute the operation task corresponding to the voice command.

[0037] In this embodiment, the vehicle-mounted mechanical arm is arranged inside the vehicle, and the vehicle-mounted mechanical arm may include a controller and a mechanical arm. In addition, other hardware devices may also be arranged inside the vehicle, such as a camera and a microphone.

[0038] See also Figure 2 , Figure 2 It is a schematic diagram of the connection of in-vehicle hardware devices provided by an embodiment of the present invention.

[0039] In this embodiment, the camera group includes at least two cameras. When the number of cameras is 2, a binocular camera is provided. The binocular camera may include a first camera and a second camera. The first camera, the second camera, the microphone and the robotic arm may be connected to the controller respectively.

[0040] See also Figure 3 , Figure 3 It is a schematic diagram of the layout position of the vehicle-mounted mechanical arm provided in an embodiment of the present invention.

[0041] A first camera 311 and a second camera 312 are respectively arranged in the vehicle on both sides of the vehicle A-pillar. The front windshield 330 is located at the front end of the vehicle, covering the entire front part of the cab, and is the main window for the driver to observe the external environment. The vehicle instrument panel 340 is located inside the cab, just below the front windshield 330, and is the installation platform for the vehicle instrument panel, central control screen and other control devices. The vehicle steering wheel 350 is located in the center of the cab, directly in front of the driver.

[0042] When installing the vehicle-mounted robotic arm 320, ensure that its positional relationship with the front windshield 330, the vehicle dashboard 340 and the vehicle steering wheel 350 is reasonable to avoid blocking the driving field of view, interfering with driving operations, and ensuring safe use.

[0043] Exemplarily, the vehicle-mounted mechanical arm 320 can be installed on the inner side of the front windshield 330, close to the driver's side. For example, in the edge area of ​​the driver's field of vision. The vehicle-mounted mechanical arm 320 can be installed above the vehicle-mounted instrument panel 320, and keep a certain distance from the vehicle-mounted instrument panel 320 to avoid interfering with the driver's viewing of instrument information. The vehicle-mounted mechanical arm 320 should be installed on the outside of the vehicle steering wheel 350, and keep a certain distance from the vehicle steering wheel 350 to ensure that the driver does not touch the bracket when operating the vehicle steering wheel 350.

[0044] The binocular camera composed of the first camera 311 and the second camera 312 acquires the actions of the driver and the co-driver in real time, and the microphone detects relevant voice data in real time. Through behavior detection and voice command detection, the movement of the vehicle-mounted robotic arm 320 is controlled to provide passengers with related actions such as object extraction, mobile phone retrieval, and water cup retrieval. In addition, the AI ​​control algorithm can be used to ensure the stability of the robotic arm during driving, such as maintaining horizontal and angle adaptation to ensure that objects such as water cups will not shake.

[0045] Furthermore, the vehicle-mounted robotic arm 320 can be designed to be retractable, hidden and foldable, and when not in use, it can be hidden in the vehicle-mounted instrument panel 320, providing an aesthetically pleasing layout.

[0046] See also Figure 4 , Figure 4 It is a schematic diagram of the structure of a vehicle-mounted mechanical arm provided in an embodiment of the present invention.

[0047] The vehicle-mounted robotic arm may include a mounting base 410 , a plurality of rotating shafts 420 , a robotic arm 430 with torque feedback, and other connecting components (not shown).

[0048] The mounting base 410 is the supporting base of the robot arm, providing a stable fixing point to ensure the stability of the robot arm during operation.

[0049] The rotating shaft 420 is a key component for the robot arm to achieve multi-degree-of-freedom motion and can rotate independently. The connecting components are connected through the joints of the rotating shaft 420 to form a motion chain of the robot arm. Optionally, each joint is equipped with a high-precision encoder for real-time feedback of the angle position of the joint, thereby achieving precise motion control.

[0050] In this embodiment, the plurality of rotating shafts 420 include one shaft, two shafts, three shafts and four shafts respectively, and the shafts are connected by joints to achieve complex movements.

[0051] In this embodiment, a manipulator 430 with torque feedback is provided at the end of the vehicle-mounted manipulator. The torque feedback function enables the manipulator to sense and control the force and torque applied to the end effector, thereby achieving more precise operation. Torque feedback can be achieved through a high-precision torque sensor, which can measure the changes in force and torque in real time.

[0052] See also Figure 5-Figure 6 , Figure 5 is a schematic diagram of a vehicle-mounted mechanical arm grabbing an object provided by an embodiment of the present invention, Figure 6 It is a schematic diagram of the folding of the vehicle-mounted mechanical arm provided by an embodiment of the present invention.

[0053] Based on this, in this embodiment, by combining the in-vehicle image data obtained by the camera group and the voice commands obtained by the microphone, the function of the vehicle-mounted robotic arm performing corresponding operation tasks according to the user's voice can be realized.

[0054] Specifically, the acquired in-vehicle image data contains multiple in-vehicle entities, which may be drivers, passengers, objects, etc. in the vehicle, and are the objects of subsequent analysis and processing.

[0055] At least two cameras are used to form a camera group to shoot the interior of the car in real time to obtain image data inside the car. Multiple cameras can capture the scene inside the car from different angles, providing more comprehensive and rich information, which is helpful for the subsequent accurate identification and positioning of entities inside the car. In addition, since the images taken by multiple cameras may have time differences, this embodiment can ensure that the image data of different cameras are consistent in time through time frame alignment and synchronization, so that subsequent feature fusion and analysis can be accurately performed.

[0056] In this embodiment, a feature extraction model is used to fuse and extract features from the in-vehicle image data. Specifically, the hybrid attention network module is the core part of the feature extraction model, and its function is to fuse and extract the image features of at least two cameras. Images taken by different cameras have different perspectives and features. The hybrid attention network module can automatically focus on important feature areas in the image and effectively fuse these features to obtain a more comprehensive and accurate feature representation.

[0057] Based on the fusion and extraction of image features, the coordinate change matrix is ​​used to locate the moving position of the vehicle-mounted robotic arm. The coordinate change matrix can convert the feature information in the image into actual physical coordinates, thereby determining the target moving position of the vehicle-mounted robotic arm in the vehicle. The feature extraction model ultimately outputs a position information map, which contains the position information of each entity in the vehicle, which provides the basis for the subsequent control of the vehicle-mounted robotic arm.

[0058] In addition, this embodiment obtains the user's voice command through the microphone, and the command reflects the operation task that the user wants the vehicle-mounted mechanical arm to perform. This embodiment controls the vehicle-mounted mechanical arm to perform the operation task corresponding to the voice command based on the position information map output by the feature extraction model and the voice command obtained by the microphone. For example, if the voice command is "pick up the cup on the seat", the position of the cup can be determined based on the position information map, and then the vehicle-mounted mechanical arm can be controlled to move to that position and perform the operation of picking up the cup.

[0059] As described above, the embodiment of the present invention provides a method for controlling a vehicle-mounted robotic arm. The use of multiple cameras and feature fusion technology can improve the recognition and positioning accuracy of entities in the vehicle, and enhance the reliability and stability of the system. Combined with voice command control, the operation of the vehicle-mounted robotic arm is more intelligent and humanized. Users can conveniently control the robotic arm through voice to complete various tasks, such as picking up items, etc., which improves the convenience and comfort of riding.

[0060] In some embodiments, the at least two cameras include a first camera and a second camera; the in-vehicle image data includes first in-vehicle image data acquired by the first camera and second in-vehicle image data acquired by the second camera; and the in-vehicle image data is input into the feature extraction model in a time frame alignment synchronization manner, including: Determine a synchronization deviation between two frames of data in the first in-vehicle image data and the second in-vehicle image data according to a hardware synchronization signal between the first camera and the second camera; If the synchronization deviation is less than or equal to a preset deviation threshold, the first in-vehicle image data and the second in-vehicle image data are input into a feature extraction model for feature extraction; If the synchronization deviation is greater than the preset deviation threshold, each frame signal of the first camera and the second camera is obtained for similarity matching. When the similarity value is greater than the preset similarity threshold, the first in-vehicle image data and the second in-vehicle image data are input into the feature extraction model for feature extraction; when the similarity value is less than or equal to the preset similarity threshold, it is determined that the synchronization has failed, and new first in-vehicle image data is collected again through the first camera, and new second in-vehicle image data is collected through the second camera for time frame alignment synchronization.

[0061] In some embodiments, the feature extraction model performs feature extraction, channel dimension reduction, and data preprocessing on the first in-vehicle image data and the second in-vehicle image data, respectively, to obtain a first feature data block and a second feature data block of the same specification; the first feature data block and the second feature data block are data-fused through a hybrid attention network module to obtain a fused feature block; a position information map is determined based on the fused feature block, and the coordinates of the vehicle-mounted robotic arm are determined through the position information map and the coordinate change matrix to locate the moving position of the vehicle-mounted robotic arm.

[0062] Exemplarily, this embodiment designs a binocular camera feature fusion deep learning network based on hybrid attention. The data captured by multiple cameras are simultaneously sent to the feature extraction model using a time frame alignment synchronization method. First, the three-channel image is extracted for features and then transformed into a low-dimensional feature block. The features of the two cameras are fused and extracted through the hybrid attention network module, and then regressed to the posture points. Finally, the coordinate change matrix is ​​used to accurately locate the moving position of the manipulator.

[0063] The input is 3-channel image data, which is first input into the feature extraction model for feature extraction. This step will extract representative feature information from the image. Then the extracted features are converted into low-dimensional feature blocks. Dimensionality reduction can reduce the complexity of the data and the amount of calculation, while retaining the key feature information of the image and improving the efficiency of subsequent processing.

[0064] The hybrid attention network module is used to fuse the features extracted by the two cameras. The hybrid attention mechanism can more effectively fuse the feature information of different cameras, enhance the ability to express object features, and thus improve the accuracy of subsequent positioning.

[0065] In the frame synchronization alignment method, this embodiment adopts a synchronization algorithm based on feature matching and hardware camera synchronization signal coordination, that is, first determine the synchronization deviation of two frames of data according to the hardware synchronization signal. If the hardware synchronization time is less than 10 nanoseconds deviation, use this data for calculation. If it is greater than 10ns and less than 1ms or the hardware synchronization signal is lost, obtain each frame signal of multiple cameras for similarity matching. Signals with lower similarity are considered to have failed synchronization, discard the data, and re-collect the signal to ensure the reliability of the system.

[0066] Therefore, when the hardware synchronization time deviation is small (less than 10 nanoseconds), it means that the frame data collected by each camera is basically synchronized in time, and these data can be used directly for subsequent calculations. This is because in this case, the time consistency of the data is high, which can ensure the accuracy of the calculation results. When the synchronization deviation is between 10 nanoseconds and 1 millisecond, or no hardware synchronization signal is received at all, the hardware synchronization effect is not good. At this time, the system will use the feature matching method for synchronization. The specific method is to obtain the features of each frame signal of multiple cameras, and then calculate the similarity between them. If the similarity between a frame signal and other signals is low, it is considered that the synchronization of the frame signal has failed. This is because under normal synchronization, the frame data collected by each camera at the same time should have a high degree of similarity. For data that fails to synchronize, the system will discard it and re-collect the signal to ensure that the data used in the end is synchronized accurately.

[0067] See also Figure 7 , Figure 7 It is a schematic diagram of a network architecture for feature extraction based on multiple cameras provided in an embodiment of the present invention.

[0068] Feature extraction and normalization are performed on the data from camera 1 and camera 2 respectively to obtain the corresponding low-dimensional feature blocks. The features of the two cameras are then fused and extracted through the hybrid attention network module, and then regressed to the posture point (POSE). Finally, the coordinate change matrix is ​​used to obtain the coordinates of the robotic arm to accurately locate the moving position of the robotic arm.

[0069] Exemplarily, an MLP-Mixer can be used to implement feature extraction and information fusion. First, the input camera data is scaled and channel preprocessed, and then the received data is encoded in different channels and sequential positions through a position encoding module. After that, parameter learning is fed back through the encoding layer and the fully connected layer. Sigmoid and quantization are used as the output of the encoder to obtain the posture point map of all objects in all images.

[0070] The input camera data is resized and channel preprocessed. Resizing can unify images of different resolutions to the same size for subsequent processing; channel preprocessing can adjust the channel information of the image to improve the effect of feature extraction. The preprocessed data is passed through the position encoding module to achieve encoding of different channels and sequential positions. Position encoding can give position information to features, allowing the model to perceive the positional relationship of features in the image, which helps to improve the model's understanding of object position and spatial structure.

[0071] In addition, the encoding layer and the fully connected layer are used for feedback parameter learning. These two layers will continuously adjust the model parameters according to the input data and preset targets (such as posture points), so that the model can better fit the data and improve the accuracy of prediction.

[0072] In this embodiment, the Sigmoid function and quantization operation are used as the output of the encoder to finally obtain the posture point map of the objects in all images. The Sigmoid function can map the output value to the [0, 1] interval, and the quantization operation can convert the continuous value into a discrete representation, which is convenient for subsequent processing and application.

[0073] In some embodiments, before the in-vehicle image data is input into the feature extraction model in a time frame alignment and synchronization manner, it also includes: calibrating the camera group and the vehicle-mounted robotic arm to establish a motion coordinate system; and determining the coordinate change matrix based on the motion coordinate system.

[0074] Calibration is the basis for the accurate operation of the entire system. After the camera group and the vehicle-mounted robot arm are installed, their actual position and posture may deviate from the theoretical value, and the relative position relationship between different devices also needs to be accurately determined. Through calibration, these deviations can be eliminated, allowing the system to accurately acquire and process image data and convert it into accurate motion control instructions for the vehicle-mounted robot arm.

[0075] Exemplarily, by calibrating the binocular camera and the robotic arm, a motion coordinate system is established, the camera is distorted and calibrated with the real space coordinates using a black and white chessboard, and the robotic arm is simultaneously moved to a standard position for hand-eye synchronization calibration. This embodiment adopts a hand-eye separation solution, which can make full use of the visual coverage of the camera layout position to provide better in-vehicle space detection.

[0076] Specifically, the calibration calculation method is: .

[0077] in, is the transformation matrix from the target coordinate system to the base coordinate system, is the transformation matrix from the target coordinate system to the camera coordinate system, is the transformation matrix from the camera coordinate system to the robot end coordinate system, It is the transformation matrix from the robot end coordinate system to the base coordinate system.

[0078] In some embodiments, according to the position information map and the voice command obtained through the microphone, the vehicle-mounted mechanical arm is controlled to perform the operation task corresponding to the voice command, including: Acquire voice commands through a microphone; parse the voice commands to acquire a target object and a target position; the target object is determined from a plurality of in-vehicle entities according to the voice commands; plan a motion path for a vehicle-mounted robotic arm to move the target object to the target position according to the target object, the target position and the position information map; and control the vehicle-mounted robotic arm to perform an operation task of moving the target object to the target position according to the motion path.

[0079] In some embodiments, the plurality of in-vehicle entities include a human body; the position information map includes human body posture information; the human body posture information includes hand posture information and / or face posture information; and the vehicle-mounted robotic arm control method further includes: According to the hand posture information and motion path, control the vehicle-mounted robotic arm to move the target object from the hand to the target position; or, control the vehicle-mounted robotic arm to move the target object from the original position to the hand as the target position; and / or, obtain the target object and target position according to the face posture information and voice commands.

[0080] Human body posture information can be obtained through human body posture point recognition. Human body posture point recognition technology can accurately detect and locate key parts or joints of the human body in images or video sequences, such as the 22 posture points of the human body. These key points may include the head, shoulders, elbows, wrists, hips, knees and ankles.

[0081] Hand posture information can be obtained through hand posture recognition. Hand posture recognition technology can accurately detect and determine the position, posture and shape of the hand from images, videos or sensor data. For example, in 21-point hand posture recognition, these key points may include finger length, width, joint angle, etc.

[0082] The face posture information can be obtained through face posture point recognition. Face posture point recognition technology can accurately locate the key feature points of the face from the face image or video frame. These feature points can fully describe the geometric shape and structural information of the face, such as face 106 posture point recognition. These key points may include contours and facial features, such as cheeks, chin, inner corners of the eyes, outer corners of the eyes, eyelids, pupil center, eyebrows, eyebrow peaks, eyebrow tails, nose roots, nose tips, nose wings, mouth corners, upper and lower lip edges, etc.

[0083] Specifically, the binocular camera captures images inside the car in real time and transmits the images to the controller. The controller uses a pre-made and trained deep algorithm model. The deep algorithm model outputs the key point information of the face, body, and hand posture end-to-end, and through the hand-eye calibration of the robotic arm, the matching of the robotic arm's moving coordinates and the real-time spatial moving points is achieved.

[0084] Furthermore, the present embodiment can also be designed with various interactive modes such as intelligent avoidance, gesture control, target following, line of sight tracking, voice interaction, etc., and cooperate with in-vehicle voice recognition to realize various interactive functions. For example, if you say "help me get a water cup", the robotic arm can quickly take the water cup for the person in the car to use. For example, carrying a bag, retrieving soup and other items, all have a good user experience.

[0085] In some embodiments, in the face posture application function, the fatigue state can be judged according to the detected facial posture, and some refreshing equipment can be configured at the position of the manipulator to help the driver reduce fatigue. At the same time, eye tracking can be achieved according to the movement of the eyes, so that the manipulator moves to the corresponding position to assist in part of the work.

[0086] In some embodiments, a series of hand-controlled tidying and actions can be set according to the detected key point information of the hand, so as to realize a series of functions such as taking items, taking water cups, taking and putting bags, etc.

[0087] As described above, the embodiment of the present invention provides a cascade positioning solution for object posture and robotic arm based on binocular cameras, which can be conveniently used for interactive services for people in the car. This embodiment uses a binocular camera feature fusion deep learning network based on hybrid attention, and sends the multi-camera shooting data into the feature extraction model after time frame alignment and synchronization, extracts features from the 3-channel image and converts it into a low-dimensional feature block. The two camera features are fused through the hybrid attention network module to regress the fine posture points of the face, body, and hand, and then accurately locate the moving position of the robotic arm with the help of the coordinate change matrix. In addition, this embodiment also realizes an intelligent robotic arm that can be folded in the space inside the car, which can assist people in the car to pick up items, carry bags, etc., and also designs functional modes such as intelligent avoidance, target following, line of sight tracking, and voice interaction.

[0088] The present invention also provides a vehicle-mounted robotic arm control device. The vehicle-mounted robotic arm control device provided by the present invention is described below. The vehicle-mounted robotic arm control device described below and the vehicle-mounted robotic arm control method described above can refer to each other.

[0089] See also Figure 8 , Figure 8 The vehicle-mounted mechanical arm control device is provided in the vehicle, and the vehicle-mounted mechanical arm control device may include an in-vehicle image data acquisition module 810 , a feature extraction model module 820 and a mechanical arm control module 830 .

[0090] The in-vehicle image data acquisition module 810 is used to acquire in-vehicle image data through real-time shooting of a camera group; wherein the camera group includes at least two cameras; and the in-vehicle image data includes multiple in-vehicle entities.

[0091] The feature extraction model module 820 is used to input the in-vehicle image data into the feature extraction model in a time frame aligned and synchronized manner to obtain a position information map output by the feature extraction model; wherein the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted robotic arm in combination with the coordinate change matrix.

[0092] The robot arm control module 830 is used to control the vehicle-mounted robot arm to perform the operation task corresponding to the voice command according to the position information map and the voice command obtained through the microphone.

[0093] In some embodiments, the at least two cameras include a first camera and a second camera; the in-vehicle image data includes first in-vehicle image data acquired by the first camera and second in-vehicle image data acquired by the second camera; the feature extraction model module 820 can be specifically used for: The synchronization deviation of two frames of data in the first in-vehicle image data and the second in-vehicle image data is determined according to the hardware synchronization signal between the first camera and the second camera; if the synchronization deviation is less than or equal to the preset deviation threshold, the first in-vehicle image data and the second in-vehicle image data are input into the feature extraction model for feature extraction; if the synchronization deviation is greater than the preset deviation threshold, each frame signal in the first camera and the second camera is obtained for similarity matching, and when the similarity value is greater than the preset similarity threshold, the first in-vehicle image data and the second in-vehicle image data are input into the feature extraction model for feature extraction; when the similarity value is less than or equal to the preset similarity threshold, it is determined that the synchronization has failed, and new first in-vehicle image data is collected again through the first camera, and new second in-vehicle image data is collected through the second camera for time frame alignment synchronization.

[0094] In some embodiments, the feature extraction model performs feature extraction, channel dimension reduction, and data preprocessing on the first in-vehicle image data and the second in-vehicle image data, respectively, to obtain a first feature data block and a second feature data block of the same specification; the first feature data block and the second feature data block are data-fused through a hybrid attention network module to obtain a fused feature block; a position information map is determined based on the fused feature block, and the coordinates of the vehicle-mounted robotic arm are determined through the position information map and the coordinate change matrix to locate the moving position of the vehicle-mounted robotic arm.

[0095] In some embodiments, the vehicle-mounted mechanical arm control device may further include a calibration module, which may be specifically used for: The camera group and the vehicle-mounted robotic arm are calibrated to establish a motion coordinate system; the coordinate change matrix is ​​determined based on the motion coordinate system.

[0096] In some embodiments, the robotic arm control module 830 can be specifically used to: obtain voice commands through a microphone; parse the voice commands to obtain the target object and target position; the target object is determined from multiple entities in the vehicle based on the voice commands; based on the target object, target position and position information map, the motion path of the vehicle-mounted robotic arm to move the target object to the target position; based on the motion path, control the vehicle-mounted robotic arm to perform the operation task of moving the target object to the target position.

[0097] In some embodiments, the multiple in-vehicle entities include a human body; the position information map includes human body posture information; the human body posture information includes hand posture information and / or face posture information; the robotic arm control module 830 can also be specifically used for: According to the hand posture information and motion path, control the vehicle-mounted robotic arm to move the target object from the hand to the target position; or, control the vehicle-mounted robotic arm to move the target object from the original position to the hand as the target position; and / or, obtain the target object and target position according to the face posture information and voice commands.

[0098] On the other hand, an embodiment of the present invention further provides an electronic device, see Fig. 9 , Fig. 9 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device may include a memory 920, a processor 910, and a computer program stored in the memory 920 and executable on the processor 910. When the processor 910 executes the program, the vehicle-mounted mechanical arm control method provided by the above methods is implemented.

[0099] Optionally, the electronic device may further include a communication bus 930 and a communication interface (CommunicationsInterface) 940, wherein the processor 910, the communication interface 940, and the memory 920 communicate with each other through the communication bus 930. The processor 910 may call a computer program in the memory 920 to execute a vehicle-mounted mechanical arm control method, which may include: The in-vehicle image data is acquired through real-time shooting of a camera group; wherein the camera group includes at least two cameras; the in-vehicle image data includes a plurality of in-vehicle entities; the in-vehicle image data is input into a feature extraction model in a time frame alignment and synchronization manner to obtain a position information map output by the feature extraction model; wherein the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted robotic arm in combination with a coordinate change matrix; according to the position information map and the voice commands acquired through the microphone, the vehicle-mounted robotic arm is controlled to execute the operation task corresponding to the voice commands.

[0100] In addition, the logic instructions in the above-mentioned memory 920 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vehicle-mounted robotic arm control method provided by the above methods. Its steps and principles have been introduced in detail in the above methods and will not be repeated here.

[0102] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the vehicle-mounted robotic arm control method provided by the above-mentioned methods. The steps and principles thereof have been introduced in detail in the above-mentioned methods and will not be repeated here.

[0103] Non-transitory computer-readable storage media can be any available media or data storage devices that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drives (SSDs)), etc.

[0104] As mentioned above, the present invention provides a vehicle-mounted robotic arm control method, device, equipment, storage medium and program product, the method comprising: obtaining in-vehicle image data through real-time shooting of a camera group; inputting the in-vehicle image data into a feature extraction model in a time frame alignment synchronization manner to obtain a position information map; the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted robotic arm in combination with the coordinate change matrix; according to the position information map and the voice instructions obtained through the microphone, the vehicle-mounted robotic arm is controlled to perform the operation task corresponding to the voice instruction. Through the above method, the present invention uses a camera group to shoot images from multiple perspectives, and then performs joint modeling to perform image feature fusion extraction; at the same time, speech recognition technology is integrated to realize the auxiliary application of the vehicle-mounted intelligent robotic arm.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle-mounted robotic arm control method, characterized in that: The vehicle-mounted mechanical arm is arranged inside the vehicle, and the vehicle-mounted mechanical arm control method comprises: Acquire in-vehicle image data by real-time shooting of a camera group; wherein the camera group includes at least two cameras; and the in-vehicle image data includes a plurality of in-vehicle entities; The in-vehicle image data is input into the feature extraction model in a time frame alignment synchronization manner to obtain a position information map output by the feature extraction model; wherein the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of the at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted mechanical arm in combination with a coordinate change matrix; According to the position information map and the voice command obtained through the microphone, the vehicle-mounted mechanical arm is controlled to perform the operation task corresponding to the voice command.

2. The vehicle-mounted mechanical arm control method according to claim 1, characterized in that: The at least two cameras include a first camera and a second camera; the in-vehicle image data include first in-vehicle image data acquired by the first camera and second in-vehicle image data acquired by the second camera; The step of inputting the in-vehicle image data into the feature extraction model in a time frame alignment and synchronization manner includes: Determine a synchronization deviation between two frames of data in the first in-vehicle image data and the second in-vehicle image data according to a hardware synchronization signal between the first camera and the second camera; If the synchronization deviation is less than or equal to a preset deviation threshold, inputting the first in-vehicle image data and the second in-vehicle image data into the feature extraction model for feature extraction; If the synchronization deviation is greater than the preset deviation threshold, each frame signal of the first camera and the second camera is obtained for similarity matching. When the similarity value is greater than the preset similarity threshold, the first in-vehicle image data and the second in-vehicle image data are input into the feature extraction model for feature extraction; when the similarity value is less than or equal to the preset similarity threshold, it is determined that the synchronization has failed, and new first in-vehicle image data is re-collected through the first camera, and new second in-vehicle image data is collected through the second camera for time frame alignment synchronization.

3. The vehicle-mounted mechanical arm control method according to claim 2, characterized in that: The feature extraction model performs feature extraction, channel dimension reduction and data preprocessing on the first in-vehicle image data and the second in-vehicle image data respectively to obtain a first feature data block and a second feature data block of the same specification; The first feature data block and the second feature data block are fused through the hybrid attention network module to obtain a fused feature block; the position information map is determined based on the fused feature block, and the coordinates of the vehicle-mounted robotic arm are determined through the position information map and the coordinate change matrix to locate the moving position of the vehicle-mounted robotic arm.

4. The vehicle-mounted mechanical arm control method according to claim 1, characterized in that: Before inputting the in-vehicle image data into the feature extraction model in a time frame alignment and synchronization manner, the method further includes: Calibrate the camera group and the vehicle-mounted mechanical arm to establish a motion coordinate system; The coordinate change matrix is ​​determined according to the motion coordinate system.

5. The vehicle-mounted mechanical arm control method according to any one of claims 1 to 4, characterized in that: The controlling the vehicle-mounted mechanical arm to perform an operation task corresponding to the voice command according to the position information map and the voice command obtained through the microphone includes: Acquiring the voice command through the microphone; Parsing the voice command to obtain a target object and a target position; the target object is determined from the multiple in-vehicle entities according to the voice command; Planning a motion path for the vehicle-mounted mechanical arm to move the target object to the target position according to the target object, the target position and the position information map; According to the motion path, the vehicle-mounted mechanical arm is controlled to perform an operation task of moving the target object to a target position.

6. The vehicle-mounted mechanical arm control method according to claim 5, characterized in that: The multiple in-vehicle entities include a human body; the position information map includes human body posture information; the human body posture information includes hand posture information and / or face posture information; The vehicle-mounted mechanical arm control method also includes: According to the hand posture information and the motion path, control the vehicle-mounted mechanical arm to move the target object from the hand to the target position; or control the vehicle-mounted mechanical arm to move the target object from the original position to the hand as the target position; And / or, acquiring the target object and the target position according to the facial posture information and the voice command.

7. A vehicle-mounted mechanical arm control device, characterized in that: The vehicle-mounted mechanical arm is arranged inside the vehicle, and the vehicle-mounted mechanical arm control device comprises: An in-vehicle image data acquisition module, configured to acquire in-vehicle image data through real-time shooting by a camera group; wherein the camera group includes at least two cameras; and the in-vehicle image data includes a plurality of in-vehicle entities; A feature extraction model module, used to input the in-vehicle image data into the feature extraction model in a time frame alignment synchronization manner, and obtain a position information map output by the feature extraction model; wherein the position information map includes the position information of each entity; the feature extraction model is used to fuse and extract the image features of the at least two cameras through a hybrid attention network module, and locate the moving position of the vehicle-mounted mechanical arm in combination with a coordinate change matrix; The robot control module is used to control the vehicle-mounted robot to perform an operation task corresponding to the voice command based on the position information map and the voice command obtained through the microphone.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the vehicle-mounted robotic arm control method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vehicle-mounted robotic arm control method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vehicle-mounted robotic arm control method according to any one of claims 1 to 6 is implemented.

Images