A method and system for embodied intelligent decision-making control based on visual recognition

Abstract

The present application proposes an embodied intelligent decision-making control method and system based on visual recognition, which belongs to the field of embodied intelligent technology. The method includes: collecting image data in real time while the robotic arm is performing the current task; inputting the image data into a pre-trained improved YOLO model to obtain the inference information features of the target object. The pre-trained improved YOLO model is constructed in a process including: adding an occlusion perception attention module to the backbone network of the YOLO model, wherein the mask of the occlusion perception attention module is a dynamic mask; comparing the inference information features of the target object with the features preset in the feature library. If the inference information features of the target object are consistent with the features preset in the feature library, the current task is stopped and the next task is executed. The present application can accurately determine the working status of the robotic arm, meet the real-time requirements, and significantly improve the overall efficiency and reliability of the embodied intelligent system.

Description

Technical Field

[0001] The present invention belongs to the field of embodied intelligence technology, and specifically relates to an embodied intelligence decision-making control method and system based on visual recognition. Background Art

[0002] In today's rapidly developing world, embodied intelligence, a cutting-edge field in artificial intelligence (AI) with both immense potential and significant challenges, is attracting the attention of researchers worldwide. It aims to empower intelligent entities with the ability to perceive, make decisions, and act in real physical environments. This requires them to understand complex and dynamic environmental information, make appropriate decisions based on this information, and execute precise actions, enabling efficient interaction with their surroundings.

[0003] Among the many technical branches of embodied intelligence, imitation learning is crucial. It allows intelligent entities to learn to perform specific tasks by observing human demonstrations or high-quality examples. However, imitation learning currently has significant problems, with ideal accuracy difficult to achieve and frequent errors. Taking industrial manufacturing as an example, when robotic arms use imitation learning to perform complex assembly tasks, they often encounter problems such as deviations in the assembly position and disordered order of parts due to misunderstandings of operational details or environmental interference. When service robots imitate humans in delivering items, they are also prone to misjudging the target location or dropping items when grasping them. These errors seriously affect the reliability and stability of embodied intelligence systems, hindering their widespread application in real-world scenarios.

[0004] For embodied intelligent systems, the ability to determine whether a model has successfully completed a given action is crucial. In an embodied intelligence architecture, the ability to drive behavioral execution is similar to the human cerebellum, precisely regulating the body movements of intelligent entities to achieve the intended actions; the ability to drive judgment and decision-making is similar to the human brain, making reasonable decisions based on environmental information and task objectives to guide the behavior of intelligent entities. Efficient and reliable embodied intelligent systems require close and precise collaboration between the brain and cerebellum. However, due to the lack of an effective action completion judgment mechanism, even if the cerebellum can prompt intelligent entities to perform actions, it cannot determine whether the actions meet the intended goals, greatly weakening system performance.

[0005] Take the use of robotic arms in industrial production as an example. On production lines, robotic arms must perform complex operations such as material handling and parts processing. Existing models for manipulating robotic arms to perform these operations struggle to accurately determine whether the operation was successfully completed. For example, after a part is installed in a designated location, the model cannot determine whether the part is correctly and securely installed. This can lead to quality issues in subsequent production steps and even equipment failure. This uncertainty is amplified in large-scale production, seriously affecting production efficiency and product quality. Embodied intelligence systems suffer from deficiencies in accuracy and slow response when determining action completion. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, this application proposes an embodied intelligent decision-making control method and system based on visual recognition.

[0007] In the first aspect, this application proposes an embodied intelligent decision-making and control method based on visual recognition, comprising:

[0008] Step S1: The robotic arm performs tasks in sequence according to the pre-set imitation learning instruction sequence;

[0009] Step S2: collecting image data in real time while the robotic arm is performing the current task;

[0010] Step S3: inputting the image data into a pre-trained improved YOLO model to obtain inference information features of the target object, wherein the pre-trained improved YOLO model is constructed by adding an occlusion-aware attention module to the backbone network of the YOLO model, wherein the mask of the occlusion-aware attention module is a dynamic mask;

[0011] Step S4: Compare the inference information features of the target object with the preset features in the feature library. If the inference information features of the target object are consistent with the preset features in the feature library, stop the current task and return to step S1 to execute the next task.

[0012] The dynamic mask is calculated as follows:

[0013] ;

[0014] Among them, M is the dynamic mask, X is the image data, is the first neural network, is the sigmoid activation function.

[0015] The first neural network is an N-layer neural network, and the size of the convolution kernel of each layer of the neural network is: ,in, is the length of the image data, is the width of the image data, N is an integer greater than 3, and the training process of the first neural network includes: using the occluded image data as samples and the target mask as a label to train to obtain the first neural network.

[0016] The pre-trained improved YOLO model construction process also includes: a feature extraction network and a feature enhancement network;

[0017] The feature extraction network is used to extract features from the output of the backbone network to obtain red channel features, green channel features, and blue channel features;

[0018] The feature enhancement network is used to calculate the covariance matrix of the three-channel features based on the red channel features, the green channel features, and the blue channel features, use the red channel features, the green channel features, the blue channel features, and the covariance matrix of the three-channel features as feature input data, reduce the dimension of the feature input data through the second neural network, and obtain first channel features and second channel features; and calculate the inference information features of the target object based on the first channel features and the second channel features.

[0019] The second neural network is a convolutional network with a convolution kernel size of .

[0020] The inference information feature of the target object is calculated based on the first channel feature and the second channel feature, and the calculation formula is as follows:

[0021] ;

[0022] in, is the inference information feature of the target object, i is the i-th channel, i is 1 or 2, is the weight parameter of the i-th channel, is the eigenvalue of the i-th channel, and b is the bias parameter.

[0023] The pre-trained improved YOLO model construction process also includes:

[0024] Step S100: Calculating the light intensity of the image data according to the image data, and selecting a dynamic threshold of the loss function of the corresponding improved YOLO model according to the light intensity;

[0025] Step S101: when the loss function of the improved YOLO model is less than the selected dynamic threshold, stop training the improved YOLO model, output the final improved YOLO model, and use the final improved YOLO model as the pre-trained improved YOLO model;

[0026] Step S102: When the loss function of the improved YOLO model is greater than or equal to the selected dynamic threshold, the number of samples is increased, the improved YOLO model is rebuilt, and the process returns to step S100;

[0027] Among them, the loss function of the improved YOLO model is the weighted sum of the mean square error and the absolute error, and the calculation formula is as follows:

[0028] ;

[0029] in, is the loss function of the improved YOLO model, α is a weight parameter between 0 and 1, is the mean square error between the sample and the label, is the absolute error between the sample and the label.

[0030] Secondly, this application proposes an embodied intelligent decision-making control system based on visual recognition, comprising:

[0031] The task execution module is used for the robotic arm to perform tasks in sequence according to the pre-set imitation learning instruction sequence;

[0032] The image acquisition module is used to collect image data in real time while the robot arm is performing the current task;

[0033] A model inference module is used to input image data into a pre-trained improved YOLO model to obtain inference information features of the target object. The pre-trained improved YOLO model is constructed by adding an occlusion-aware attention module to the backbone network of the YOLO model, wherein the mask of the occlusion-aware attention module is a dynamic mask;

[0034] The visual recognition module is used to compare the inference information features of the target object with the preset features in the feature library. If the inference information features of the target object are consistent with the preset features in the feature library, the current task is stopped and the task execution module is returned to execute the next task.

[0035] On the third aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the embodied intelligent decision-making control method based on visual recognition.

[0036] In a fourth aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to execute the embodied intelligent decision-making control method based on visual recognition.

[0037] Beneficial effects:

[0038] This application proposes an embodied intelligent decision-making and control method and system based on visual recognition. Compared with the traditional method of determining the completion of an action, it can accurately determine the working status of the robotic arm, effectively make up for the defects of the existing embodied intelligent system in determining the completion of an action, such as insufficient accuracy and slow response, and significantly improve the overall performance and reliability of the embodied intelligent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of an embodied intelligent decision-making control method based on visual recognition according to an embodiment of the present application;

[0040] Figure 2 This is a flow chart of an embodied intelligent decision-making control method based on visual recognition according to an embodiment of the present application;

[0041] Figure 3 This is a principle block diagram of an embodied intelligent decision-making control system based on visual recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The specific implementation of the present application is further described in detail below with reference to the accompanying drawings and examples.

[0043] To address the technical issues presented in the background art, visual recognition technology, as a powerful perception tool, can provide rich and accurate information for embodied intelligence system decision-making. Through visual recognition, the system can obtain key information such as the position, shape, and posture of objects in the environment in real time, thereby more accurately determining whether the current action is achieving the intended goal. For example, using visual recognition technology to monitor the assembly status of parts during robotic arm operation can promptly detect problems such as position deviations and unstable installation, providing a reliable basis for subsequent decision-making. Therefore, decision-making based on visual recognition methods has become a key approach to solving the current difficulties of embodied intelligence systems and improving the overall efficiency and reliability of the system. It is expected to promote the widespread application of embodied intelligence in many fields, such as industry, healthcare, and services. However, because visual recognition technology requires the application of model algorithms, complex models with large data volumes typically require a huge amount of computation. Furthermore, the real-time nature of each task performed by the robotic arm requires attention. Balancing accuracy and time consumption also presents a significant challenge.

[0044] Example 1:

[0045] This embodiment proposes an embodied intelligent decision-making control method based on visual recognition, such as Figure 1 、 Figure 2 Shown, including:

[0046] Step S1: The robotic arm performs tasks in sequence according to the pre-set imitation learning instruction sequence;

[0047] In this embodiment, the initialization phase begins, in which the entire robotic arm task execution and determination system is activated, completing system initialization. This includes loading the parameters of the pre-trained improved YOLO model, initializing visual acquisition devices such as cameras, establishing communication connections with the robotic arm, and ensuring that all components are in a working state. This step is prior art and will not be further described in this application.

[0048] After the start phase, this embodiment enters the task execution phase, and the robotic arm begins to perform tasks according to pre-set imitation learning strategies or instructions, such as grasping and installing parts in industrial assembly scenarios, grasping and delivering items in service scenarios, etc.

[0049] Step S2: collecting image data in real time while the robotic arm is performing the current task;

[0050] In this embodiment, while the robotic arm is performing a task, visual acquisition devices such as cameras deployed in the work scene collect image data in real time, including: capturing the robotic arm operation process and the status information of the target object.

[0051] In actual application scenarios, advanced visual acquisition equipment, such as high-resolution cameras, is deployed across the robotic arm's work area from all angles and in all directions. These devices possess exceptional image capture capabilities, clearly recording subtle movements and object features. Furthermore, they can flexibly adjust to factors such as ambient lighting and spatial layout, ensuring the capture of complete and accurate image data, providing a solid foundation for subsequent target recognition.

[0052] Step S3: inputting the image data into a pre-trained improved YOLO model to obtain inference information features of the target object, wherein the pre-trained improved YOLO model is constructed by adding an occlusion-aware attention module to the backbone network of the YOLO model, wherein the mask of the occlusion-aware attention module is a dynamic mask;

[0053] In this embodiment, the collected image data is input into the pre-trained improved YOLO model for inference operation. The model identifies the target object in the image and analyzes its key information such as position, shape, and posture, among which the position, shape, and posture are the inference information features of the target object.

[0054] In the embodied intelligent decision-making and control method, in order to balance the relationship between accuracy and time consumption, avoid the problems of insufficient precision and slow response in the existing technology, and solve the technical difficulties of accurate recognition and task completion judgment in an occluded environment, this embodiment improves the existing YOLO model. This embodiment uses the YOLOv8 model (You Only Look Once version 8). Other versions of the YOLO model can also be modified accordingly to achieve the effect achieved by the pre-trained improved YOLO model mentioned in this embodiment. However, it has been verified that other versions of the YOLO model are not as good as the modified YOLOv8 model. The improved YOLO model can achieve high-precision visual recognition of the robot arm operation process and the status information of the target object.

[0055] In this embodiment, the YOLOv8 model is improved in three aspects, so that the method of this embodiment can accurately perform visual recognition even in an occluded environment. In the improvement of the algorithm, attention is paid to using as little computation as possible to meet real-time requirements.

[0056] The first aspect of improvement:

[0057] This implementation embeds the Occlusion-Aware Attention (OAA) module of ORCTrack (Occlusion-Aware Detection and Re-ID Calibrated Network for Multi-Object Tracking) within the Convolutional Block with Shortcut (CBS) module of the YOLOv8 model's backbone network. This module leverages ORCTrack's strengths in occlusion scene processing with YOLOv8's efficient object detection architecture, effectively enhancing the adaptive extraction of spatial features at each convolutional stage. Leveraging high-order statistical features from the overall representation, this module emphasizes spatial details in feature channels, focusing on visible foreground objects while suppressing interference from occluded background regions.

[0058] In practice, occlusion is common, but the masks used in previous target recognition methods are mostly static. This embodiment proposes a dynamic mask (VAM, Vehicle Allocation Matrix) that continuously changes based on the target's real-time status and context. This allows the model to more accurately focus on foreground targets in various occlusion scenarios, enhancing the model's adaptability to complex environments. The dynamic mask is generated by the first neural network using the following formula:

[0059] ;

[0060] Among them, M is the dynamic mask, X is the image data, is the first neural network, is the sigmoid activation function.

[0061] The first neural network is an N-layer neural network, and the size of the convolution kernel of each layer of the neural network is: ,in, is the length of the image data, is the width of the image data, N is an integer greater than 3, and the training process of the first neural network includes: using the occluded image data as samples and the target mask as a label to train to obtain the first neural network.

[0062] The second improvement:

[0063] The YOLO model not only has a backbone network, but also a feature extraction network. In this embodiment, a feature enhancement network is added after the feature extraction network, so that the inference information features of the target object can be extracted more accurately while meeting the real-time requirements.

[0064] The pre-trained improved YOLO model construction process also includes: a feature extraction network and a feature enhancement network;

[0065] The feature extraction network is used to extract features from the output of the backbone network to obtain red channel features R, green channel features G, and blue channel features B;

[0066] The feature enhancement network is used to calculate the covariance matrix C of the three-channel features based on the red channel features R, the green channel features G, and the blue channel features B, and use the red channel features R, the green channel features G, the blue channel features B, and the covariance matrix C of the three-channel features as feature input data. The feature input data is reduced in dimension through the second neural network to obtain the first channel features X1 and the second channel features X2; based on the first channel features X1 and the second channel features X2, the inference information features Y of the target object are calculated.

[0067] The second neural network is a convolutional network with a convolution kernel size of .

[0068] The inference information feature of the target object is calculated based on the first channel feature and the second channel feature, and the calculation formula is as follows:

[0069] ;

[0070] in, is the inference information feature of the target object, i is the i-th channel, i is 1 or 2, is the weight parameter of the i-th channel, is the eigenvalue of the i-th channel, and b is the bias parameter. In this embodiment, the value range of b is: [0.01, 0.1], The value range of the weight parameter is: [-0.414, 0.414].

[0071] In this embodiment, in terms of feature processing, feature enhancement based on the covariance matrix is ​​adopted. After adjusting the number of channels using 1×1 convolution, the correlation between different channel features is reflected by calculating the covariance matrix, where the covariance matrix is ​​calculated as follows:

[0072] ;

[0073] in, is the covariance between the red channel feature R and the green channel feature G, is the covariance between the red channel feature R and the blue channel feature B, is the covariance between the green channel feature G and the blue channel feature B.

[0074] In this embodiment, in order to more accurately extract the inference information features of the target object, a feature enhancement network is added after the feature extraction network of the YOLO model. After feature extraction of the output of the backbone network, the features of three channels are obtained, including: red channel features R, green channel features G, and blue channel features B; based on these three features, the covariance matrix C of the three-channel features is calculated, and the red channel features R, green channel features G, blue channel features B, and the covariance matrix C of the three-channel features are used as feature input data. The feature input data is reduced in dimension through 1×1 convolution to obtain the first channel features X1 and the second channel features X2; based on the first channel features X1 and the second channel features X2, the inference information features Y of the target object are calculated. Through the above operations, more accurate inference information features of the target object are obtained. At the same time, the computational complexity of the above steps is not large, which can meet the real-time requirements.

[0075] The third improvement:

[0076] The pre-trained improved YOLO model construction process also includes:

[0077] Step S100: Calculating the light intensity of the image data according to the image data, and selecting a dynamic threshold of the loss function of the corresponding improved YOLO model according to the light intensity;

[0078] Step S101: when the loss function of the improved YOLO model is less than the selected dynamic threshold, stop training the improved YOLO model, output the final improved YOLO model, and use the final improved YOLO model as the pre-trained improved YOLO model;

[0079] Step S102: When the loss function of the improved YOLO model is greater than or equal to the selected dynamic threshold, the number of samples is increased, the improved YOLO model is rebuilt, and the process returns to step S100;

[0080] Among them, the loss function of the improved YOLO model is the weighted sum of the mean square error and the absolute error, and the calculation formula is as follows:

[0081] ;

[0082] in, is the loss function of the improved YOLO model, α is a weight parameter between 0 and 1, is the mean square error between the sample and the label, is the absolute error between the sample and the label.

[0083] In this embodiment, in the fields of object detection and computer vision, the quality of solving regression problems is directly related to the model's accuracy in predicting object attributes such as location and size. Traditional loss functions, such as mean squared error (MSE) and mean absolute error (MAE), while performing well in many scenarios, are sensitive to outliers. This can cause the model to be overly influenced by a small number of outliers during training, thereby reducing overall generalization ability and prediction accuracy. To overcome this problem, this embodiment proposes the Dynamic Thresholded Intersection over Union (DTIOU) loss function. This function not only combines the advantages of mean squared error (MSE) and absolute error (ABS), but also effectively reduces sensitivity to outliers by introducing a dynamic threshold mechanism, making the model more robust for regression problems. The dynamic threshold mechanism dynamically sets a specified threshold based on the actual light conditions identified in the real-time image data. For example, if the light conditions are poor (i.e., the light identification result is less than a first set light threshold) and there are many obstructions, the threshold is set to 0.5 to 0.99. If the light conditions are good (i.e., the light identification result is greater than or equal to a second set threshold) and there are few obstructions, the threshold is set to 0.01 to 0.49. Multiple light thresholds can also be set as needed to achieve the effect of dynamic thresholding.

[0084] The combined improvements in these three areas enable the pre-trained, improved YOLO model to accurately capture the inference information features of target objects while also meeting real-time requirements. This allows for extremely fast in-depth processing of images of the work scene as the robotic arm performs tasks. After the robotic arm completes each task, the visual recognition system analyzes the captured images within milliseconds.

[0085] The system can accurately identify the position information of the target object, whether it is millimeter-level position deviation or relative position relationship in complex space, and can accurately present it; as for the shape of the target object, even extremely complex and irregular contours can be clearly defined through advanced image recognition technology.

[0086] Step S4: Compare the inference information features of the target object with the preset features in the feature library. If the inference information features of the target object are consistent with the preset features in the feature library, stop the current task and return to step S1 to execute the next task.

[0087] In this embodiment, the visual recognition module meticulously compares key information about the identified target object with pre-defined features in a carefully constructed completion state feature library within the system. For example, in a parts assembly task, this module determines whether the part is installed in the specified position and at the correct angle. This feature library, constructed through in-depth analysis and learning from a large number of samples of successfully completed tasks, covers multi-dimensional features such as the standard position, shape, outline, and posture angle of the target object upon completion of each task.

[0088] If the comparison result meets the preset task completion criteria, the judgment is "Y" (Yes), indicating that the task was successfully completed. If the judgment result is "Y", the system sends a task completion command to the robot arm, causing it to stop the current task and proceed to the next task according to subsequent arrangements. If the comparison result does not meet the preset criteria, the judgment is "N" (No), indicating that the task was not successfully completed. If the judgment result is "Y", the system sends a task completion command to the robot arm, causing it to stop the current task and proceed to the next task according to subsequent arrangements. If the judgment result is "N", the system issues an alarm indicating that the task execution failed and sends a command to the robot arm to return to its initial state or restart the task according to a specific strategy. The model inference phase then re-enters the process, repeating the above process until the task is successfully completed.

[0089] If, after rigorous comparison, the system determines that the current working state meets the preset completion status characteristics, it will quickly and accurately send a completion command to the robotic arm via a high-speed, stable communication protocol, allowing the robotic arm to immediately receive and respond, and then proceed to the next task in an orderly manner. Conversely, if the current imitation learning process is determined to have failed to complete the task, the system will immediately trigger an alarm mechanism. This alarm will not only be prominently displayed on the local operation interface, but will also be transmitted synchronously via the network to the remote monitoring terminal, allowing relevant technicians to obtain timely information.

[0090] At the same time, the system automatically initiates the task restart process, and the robotic arm will re-execute the task according to the preset initial motion sequence. During this cycle, the system records and analyzes the data generated by each task execution, continuously optimizing the pre-trained improved YOLO model and task execution strategy, thereby gradually improving the success rate of task completion.

[0091] Compared to traditional methods for determining action completion, this embodiment proposes an embodied intelligence decision-making and control method based on visual recognition. This method can accurately determine the working state of a robotic arm, effectively overcoming the shortcomings of existing embodied intelligence systems in determining action completion, such as insufficient precision and slow response. This method will significantly improve the overall performance and reliability of embodied intelligence systems, laying a solid foundation for their widespread application in numerous complex and critical scenarios, including precision assembly in industrial production, surgical assistance in the medical field, and personalized services in the service industry. This will effectively propel embodied intelligence technology from theoretical research to large-scale practical application.

[0092] Example 2:

[0093] This embodiment proposes an embodied intelligent decision-making control system based on visual recognition, such as Figure 3 The system includes: a task execution module, an image acquisition module, a model reasoning module and a visual recognition module; the task execution module is connected to the image acquisition module, the image acquisition module is connected to the model reasoning module, the model reasoning module is connected to the visual recognition module, and the visual recognition module is connected to the task execution module;

[0094] The task execution module is used for the robotic arm to perform tasks in sequence according to the pre-set imitation learning instruction sequence;

[0095] The image acquisition module is used to collect image data in real time while the robot arm is performing the current task;

[0096] A model inference module is used to input image data into a pre-trained improved YOLO model to obtain inference information features of the target object. The pre-trained improved YOLO model is constructed by adding an occlusion-aware attention module to the backbone network of the YOLO model, wherein the mask of the occlusion-aware attention module is a dynamic mask;

[0097] The visual recognition module is used to compare the inference information features with the preset features in the feature library. If the inference information features are consistent with the preset features in the feature library, the current task is stopped and the task execution module is returned to execute the next task.

[0098] Example 3:

[0099] This embodiment proposes an electronic device, comprising: one or more processors, and a memory, wherein the memory is used to store instructions. When the instructions are executed by the one or more processors, the one or more processors execute the embodied intelligent decision-making control method based on visual recognition.

[0100] The electronic device can be a mobile phone, computer, or tablet computer, and includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the embodied intelligent decision-making and control method based on visual recognition as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0101] The processor is configured to execute all or part of the steps of the embodied intelligent decision-making and control method based on visual recognition as described in the above embodiment. The memory is configured to store various types of data, such as instructions for any application or method in the electronic device, as well as data related to the application.

[0102] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute the embodied intelligent decision-making control method based on visual recognition described in the above embodiment.

[0103] Example 4:

[0104] This embodiment provides a computer-readable storage medium storing executable instructions. When the instructions are executed, if they are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0105] The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of an embodied intelligent decision-making control method based on visual recognition described in various embodiments of the present application.

[0106] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, APP (Application, abbreviation of application software) application store and other media that can store program verification codes, on which computer programs are stored. When the computer program is executed by the processor, it can implement the various steps of the above-mentioned embodied intelligent decision-making and control method based on visual recognition.

[0107] The various embodiments in this application are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0108] The scope of protection of this application is not limited to the above-described embodiments. Obviously, those skilled in the art may make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, the disclosure is intended to include such modifications and variations.

Claims

1. A method for embodied intelligent decision-making and control based on visual recognition, characterized in that: include: Step S1: The robotic arm performs tasks in sequence according to the pre-set imitation learning instruction sequence; Step S2: collecting image data in real time while the robotic arm is performing the current task; Step S3: inputting the image data into a pre-trained improved YOLO model to obtain inference information features of the target object, wherein the pre-trained improved YOLO model is constructed by adding an occlusion-aware attention module to the backbone network of the YOLO model, wherein the mask of the occlusion-aware attention module is a dynamic mask; Step S4: Compare the inference information features of the target object with the preset features in the feature library. If the inference information features of the target object are consistent with the preset features in the feature library, stop the current task and return to step S1 to execute the next task.

2. The method of embodied intelligent decision-making and control based on visual recognition according to claim 1, characterized in that: The dynamic mask is calculated as follows: M=σ(f θ (X)); Among them, M is the dynamic mask, X is the image data, f θ () is the first neural network, and σ is the sigmoid activation function.

3. The method of embodied intelligent decision-making and control based on visual recognition according to claim 2, characterized in that: The first neural network is an N-layer neural network, and the size of the convolution kernel of each layer of the neural network is: n1×n2, where n1 is the length of the image data, n2 is the width of the image data, and N is an integer greater than 3. The training process of the first neural network includes: using the occluded image data as a sample and the target mask as a label for training to obtain the first neural network.

4. The method for embodied intelligent decision-making and control based on visual recognition according to claim 1, characterized in that: The pre-trained improved YOLO model construction process also includes: a feature extraction network and a feature enhancement network; The feature extraction network is used to extract features from the output of the backbone network to obtain red channel features, green channel features, and blue channel features; The feature enhancement network is used to calculate the covariance matrix of the three-channel features based on the red channel features, the green channel features, and the blue channel features, and use the red channel features, the green channel features, the blue channel features, and the covariance matrix of the three-channel features as feature input data. The feature input data is reduced in dimension by a second neural network to obtain first channel features and second channel features; and based on the first channel features and the second channel features, the inference information features of the target object are calculated.

5. The method for embodied intelligent decision-making and control based on visual recognition according to claim 4, characterized in that: The second neural network is a one-layer convolutional network with a convolution kernel size of 1×1.

6. The method for embodied intelligent decision-making and control based on visual recognition according to claim 4, characterized in that: The inference information feature of the target object is calculated based on the first channel feature and the second channel feature, and the calculation formula is as follows: Among them, Y is the inference information feature of the target object, i is the i-th channel, W i is the weight parameter of the i-th channel, X i is the eigenvalue of the i-th channel, and b is the bias parameter.

7. The method of embodied intelligent decision-making and control based on visual recognition according to claim 1, characterized in that: The pre-trained improved YOLO model construction process also includes: Step S100: Calculating the light intensity of the image data according to the image data, and selecting a dynamic threshold of the loss function of the corresponding improved YOLO model according to the light intensity; Step S101: when the loss function of the improved YOLO model is less than the selected dynamic threshold, stop training the improved YOLO model, output the final improved YOLO model, and use the final improved YOLO model as the pre-trained improved YOLO model; Step S102: When the loss function of the improved YOLO model is greater than or equal to the selected dynamic threshold, the number of samples is increased, the improved YOLO model is rebuilt, and the process returns to step S100; Among them, the loss function of the improved YOLO model is the weighted sum of the mean square error and the absolute error, and the calculation formula is as follows: Emix = α × MSE + (1-α) × MAE; Among them, Emix is ​​the loss function of the improved YOLO model, α is a weight parameter between 0 and 1, MSE is the mean square error between the sample and the label, and MAE is the absolute error between the sample and the label.

8. An embodied intelligent decision-making control system based on visual recognition, characterized in that: include: The task execution module is used for the robotic arm to perform tasks in sequence according to the pre-set imitation learning instruction sequence; The image acquisition module is used to collect image data in real time while the robot arm is performing the current task; A model inference module is used to input image data into a pre-trained improved YOLO model to obtain inference information features of the target object. The pre-trained improved YOLO model is constructed by adding an occlusion-aware attention module to the backbone network of the YOLO model, wherein the mask of the occlusion-aware attention module is a dynamic mask; The visual recognition module is used to compare the inference information features of the target object with the preset features in the feature library. If the inference information features of the target object are consistent with the preset features in the feature library, the current task is stopped and the task execution module is returned to execute the next task.

9. An electronic device, characterized in that: include: One or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the embodied intelligent decision-making and control method based on visual recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that It stores executable instructions, which, when executed, enable the processor to execute the embodied intelligent decision-making and control method based on visual recognition as described in any one of claims 1 to 7.

Images