Visual perception method of humanoid robot and related equipment

Abstract

The embodiment of the invention provides a visual perception method of a humanoid robot and related equipment, and belongs to the technical field of robots. The method comprises the following steps: acquiring label-free visual data and label visual data of a target scene; inputting the standardized image block data set into a self-supervised learning model, and training the unlabeled visual data based on the self-supervised learning model to extract a first visual feature of the target image block data; inputting the standardized image block data set into a self-supervised learning model, and extracting a second visual feature from the labeled visual data by using the self-supervised learning model; and based on the first visual feature and the second visual feature, training a multi-task learning model so that the humanoid robot can execute at least two different visual perception tasks at the same time, and generating and executing a corresponding action instruction according to a perception result of the multi-task learning model. According to the embodiment of the invention, the adaptability and operation continuity of the humanoid robot in a cross-task environment can be enhanced.

Description

Visual perception methods and related equipment for humanoid robots Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a visual perception method and related equipment for a humanoid robot. Background Technology

[0002] Among related technologies, humanoid robots have been widely used in industrial assembly, home services and other scenarios due to their flexibility and versatility. Visual perception is the core foundation for them to achieve complex tasks. In industrial scenarios, visual recognition is needed to identify defects in parts and guide the robotic arm to assemble them precisely. In home scenarios, visual recognition is needed to identify object categories and warn of dangerous behaviors. Summary of the Invention

[0003] The main objective of this application is to propose a visual perception method and related equipment for humanoid robots, which can enhance the adaptability and operational consistency of humanoid robots in cross-task environments.

[0004] To achieve the above objectives, one aspect of this application proposes a visual perception method for a humanoid robot. The method includes: acquiring unlabeled visual data and labeled visual data of a target scene; preprocessing the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; inputting the standardized image patch dataset into a self-supervised learning model, processing the unlabeled visual data based on the self-supervised learning model to extract a first visual feature of the target image patch data; inputting the standardized image patch dataset into the self-supervised learning model, using the self-supervised learning model to extract a second visual feature from the labeled visual data; and training a multi-task learning model based on the first visual feature and the second visual feature, so that the humanoid robot can simultaneously perform at least two different visual perception tasks, and generate and execute corresponding action instructions according to the perception results of the multi-task learning model.

[0005] In some embodiments, the preprocessing of the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset includes: normalizing the images in the unlabeled visual data and the labeled visual data to a preset size and resolution; dividing the normalized images into non-overlapping image patches, and for the image patches corresponding to the labeled visual data, calculating the proportion of the target region in the image patch based on its annotation information; and filtering out image patches whose target region proportion exceeds a preset threshold to form the standardized image patch dataset.

[0006] In some embodiments, the self-supervised learning model includes a feature extraction backbone network and a scene adaptation layer; the feature extraction backbone network is used to extract the first visual feature and the second visual feature from the target image patch data; the scene adaptation layer is used to process the first visual feature and the second visual feature to output block coding features containing spatial location information and class coding features containing global semantic information.

[0007] In some embodiments, the multi-task learning model includes at least two parallel task branches; before generating and executing corresponding action instructions based on the perception results of the multi-task learning model, the method further includes: receiving the block-encoded features through a first task branch in the multi-task learning model and outputting a first type of perception result; the first type of perception result is a pixel-level segmentation result; receiving the class-encoded features through a second task branch in the multi-task learning model and outputting a second type of perception result; the second type of perception result is a classification result or a regression estimation result.

[0008] In some embodiments, training a multi-task learning model based on the first visual feature and the second visual feature to enable a humanoid robot to simultaneously perform at least two different visual perception tasks includes: defining a dynamic loss function for the multi-task learning model; the dynamic loss function is used to jointly optimize the at least two task branches, the dynamic loss function includes at least a first loss term and a second loss term, the first loss term corresponding to the first type of perception task and the second loss term corresponding to the second type of perception task; during training, dynamically adjusting the weight coefficients of the first loss term and the second loss term in the total loss based on the first visual feature and the second visual feature to obtain the trained multi-task learning model.

[0009] In some embodiments, after training a multi-task learning model based on the first visual feature and the second visual feature to enable a humanoid robot to perform at least two different visual perception tasks simultaneously, the method further includes: evaluating the trained multi-task learning model using a preset evaluation metric; if the multi-task learning model is determined to meet preset evaluation conditions, a trained multi-task learning model is obtained; the evaluation metric includes at least the mean average accuracy for evaluating segmentation tasks, the area under the receiver operating characteristic curve for evaluating classification tasks, and the error value for evaluating regression tasks.

[0010] In some embodiments, generating and executing corresponding action instructions based on the perception results of the multi-task learning model includes: if it is determined that the perception results meet a first preset condition, generating a first type of control instruction to control the humanoid robot to perform a first type of action; if the perception results meet a second preset condition, generating a second type of control instruction to control the humanoid robot to perform a second type of action; wherein the first type of action is different from the second type of action.

[0011] To achieve the above objectives, another aspect of this application proposes a visual perception device for a humanoid robot. The device includes: a data acquisition module for acquiring unlabeled visual data and labeled visual data of a target scene; a data processing module for preprocessing the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; a first extraction module for inputting the standardized image patch dataset into a self-supervised learning model, and training the unlabeled visual data based on the self-supervised learning model to extract a first visual feature from the target image patch data; a second extraction module for inputting the standardized image patch dataset into the self-supervised learning model, and using the self-supervised learning model to extract a second visual feature from the labeled visual data; and a model training module for training a multi-task learning model based on the first and second visual features, enabling the humanoid robot to simultaneously perform at least two different visual perception tasks, and generating and executing corresponding action commands based on the perception results of the multi-task learning model.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a visual perception method, device, electronic device, storage medium, and program product for a humanoid robot. This solution acquires unlabeled and labeled visual data of a target scene; preprocesses the unlabeled and labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; the standardized image patch dataset is input into a self-supervised learning model, and the unlabeled visual data is trained based on the self-supervised learning model to extract the first visual features of the target image patch data; the standardized image patch dataset is input into the self-supervised learning model, and the labeled visual data is extracted using the self-supervised learning model; based on the first and second visual features, a multi-task learning model is trained so that the humanoid robot can simultaneously perform at least two different visual perception tasks, and generate and execute corresponding action instructions based on the perception results of the multi-task learning model. By implementing the embodiments of this application, the reliance on a large amount of labeled data is reduced through self-supervised learning, and the generalization ability and efficiency of the model in cross-task scenarios are improved by multi-task learning. At the same time, it promotes the close integration of perception and execution, which can enhance the adaptability and operational consistency of humanoid robots in cross-task environments. Attached Figure Description

[0016] Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 is a flowchart of a visual perception method for a humanoid robot provided in an embodiment of this application; Figure 3 is a schematic diagram of the architecture of visual perception for a humanoid robot in an embodiment; Figure 4 is a schematic diagram of the structure of a visual perception device for a humanoid robot provided in an embodiment of this application; Figure 5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1) DINOv2 is a self-supervised vision model based on the Vision Transformer architecture, which has powerful unsupervised feature learning capabilities and a wide range of application scenarios.

[0023] In related technologies, humanoid robots, as highly flexible and versatile intelligent agents, rely on their visual perception capabilities as a crucial foundation for achieving complex tasks such as industrial assembly and household services. However, this technology still faces several key challenges: First, training visual models relies on a large amount of high-quality labeled data. Defect segmentation and pose annotation in industrial scenarios require specialized manpower, while the annotation of dangerous behaviors in household scenarios also suffers from low efficiency and strong subjectivity, making it difficult for models to be effectively applied in data-scarce scenarios.

[0024] Secondly, existing solutions typically train dedicated models independently for different tasks such as defect detection, pose estimation, and object recognition. This not only results in inference latency that fails to meet real-time response requirements, but also a lack of feature sharing between models, leading to a significant decrease in generalization performance when scenarios or tasks change. Furthermore, current vision systems mostly remain at the perception output stage, failing to form an automatic closed loop with the robot's execution actions. This lag between perception and execution affects the smoothness of industrial operations and the speed of emergency response in home scenarios. Although deep learning has achieved some performance improvements in single vision tasks, the core problems mentioned above—high annotation costs, low efficiency of multi-task collaboration, and the disconnect between perception and execution—have not yet been systematically resolved, hindering the large-scale deployment of humanoid robots in typical scenarios such as automobile assembly and elderly care.

[0025] In view of this, this application provides a visual perception method for humanoid robots, which can enhance the adaptability and operational consistency of humanoid robots in cross-task environments.

[0026] The visual perception method for humanoid robots provided in this application relates to the field of robotics. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the visual perception method for humanoid robots, but is not limited to the above forms.

[0027] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0028] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0029] Figure 1 shows a schematic diagram of an implementation environment provided in this application. Referring to Figure 1, the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network through wireless or wired means to complete data transmission and exchange.

[0030] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0031] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0032] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the application does not impose any limitations.

[0033] For example, based on the implementation environment shown in FIG1, this application embodiment provides a visual perception method for a humanoid robot. The following description takes the application of the visual perception method of the humanoid robot in server 101 as an example. It can be understood that the visual perception method of the humanoid robot can also be applied in terminal 102.

[0034] Figure 2 is an optional flowchart of a visual perception method for a humanoid robot provided in an embodiment of this application. The execution subject of the visual perception method for the humanoid robot can be any of the aforementioned electronic devices (including servers or terminals). The method in Figure 2 may include, but is not limited to, steps S201 to S205.

[0035] Step S201: Obtain unlabeled visual data and labeled visual data of the target scene.

[0036] In some embodiments, the target scene can be the environment in which the humanoid robot is currently performing its task. Specifically, the target scene can be categorized into an industrial scenario like automotive bearing assembly and a home scenario like elderly care. Electronic devices can communicate with the humanoid robot to receive both unlabeled and labeled visual data collected by the robot. The electronic devices can also send motion commands to the humanoid robot to control it to perform task operations. Optionally, the electronic devices can systematically collect two types of visual data: unlabeled and labeled visual data. Unlabeled visual data refers to image or video streams directly from the scene without any human annotation; it is low-cost to acquire and easily scalable. Labeled visual data refers to visual samples that have been professionally labeled and contain specific task-related information.

[0037] Specifically, the hardware used includes the LiDian F1 industrial model and the LiDian D1 home model humanoid robots, each equipped with a 120fps industrial camera capable of withstanding strong light and shadow interference, and a home camera with a 120-degree wide-angle view and infrared mode, respectively. A 2TB data server supporting TFRecord format and JSON tag-associated storage is also provided. The software environment is built based on the PyTorch 2.0 deep learning framework, the CUDA 11.8 computing platform, the OpenCV 4.8 image processing library, and a dedicated robotic arm control SDK. The datasets used include the NEU-DET extended dataset for industrial scenarios, containing 5000 unlabeled automotive bearing images and 300 finely labeled images, with labels covering defect segmentation regions and assembly hole pose information; and a dataset for home scenarios, containing 3000 unlabeled environmental images and 200 labeled images collected from 20 homes, with labels covering common object segmentation and behavior classification. All data is stored in TFRecord format and associated with JSON format tag files.

[0038] Step S202: Preprocess the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data.

[0039] In some embodiments, images in both unlabeled and labeled visual data are uniformly normalized to a preset size and resolution; the normalized images are divided into non-overlapping image blocks, and for the image blocks corresponding to labeled visual data, the proportion of the target region in the image block is calculated based on its annotation information; image blocks whose target region proportion exceeds a preset threshold are selected to form a standardized image block dataset.

[0040] In the data preprocessing stage, the electronic device first normalizes all images by using OpenCV to uniformly scale them to 640×640 pixels and convert them to the RGB color space, while standardizing the resolution to 300 dpi. For unlabeled data, 5,000 images from industrial scenes and 3,000 images from home scenes are non-overlapping sliding cuts with a step size of 64×64. Industrial video frames are sampled at 10 frames per second, and redundant frames with inter-frame differences of less than 5% are filtered out. Home video frames are sampled at 5 frames per second, and frames containing moving targets are retained. Finally, a total of 800,000 image blocks are generated and stored in GZIP compressed TFRecord format. For labeled data, there are 300 labeled images in the industrial scene and 200 labeled images in the home scene. The corresponding segmented labeled images are processed by cutting them in sync with the original images. By calculating the percentage of defective pixels in each segmented block and removing low percentage image blocks based on the experimentally verified 30% threshold, 8200 valid labeled blocks from the industrial scene and 5500 from the home scene are finally retained. Each block is associated with a JSON tag file that records detailed information such as data path, label type and coordinates through a unique identifier.

[0041] By performing a series of preprocessing steps on both unlabeled and labeled visual data, the accuracy of standardized image patch datasets can be improved, thus providing a technical foundation for subsequent processing by self-supervised learning models.

[0042] Step S203: Input the standardized image patch dataset into the self-supervised learning model, and process the unlabeled visual data based on the self-supervised learning model to extract the first visual features of the target image patch data.

[0043] In some embodiments, the self-supervised learning model includes a feature extraction backbone network and a scene adaptation layer. The feature extraction backbone network extracts first visual features and second visual features from target image patch data. The scene adaptation layer processes the first visual features and second visual features to output block-encoded features containing spatial location information and class-encoded features containing global semantic information. Further, the feature extraction backbone network typically employs a deep convolutional neural network or a visual Transformer architecture. Its core function is to automatically learn and extract deep visual features from the input target image patch. These features include both first visual features derived from unlabeled data that contain the inherent structure of the data, and second visual features derived from labeled data that are associated with a specific task. Subsequently, the scene adaptation layer further processes and refines these basic visual features. By introducing spatial attention mechanisms or feature pyramids, the scene adaptation layer generates block-encoded features that finely express the local positional relationships of the image, and integrates class-encoded features that reflect the overall content and contextual semantics of the image.

[0044] Specifically, a self-supervised learning model can be constructed based on the DINOv2 vit-base-patch16 model, with a scene adaptation layer added after the encoder. This layer contains two 1x1 convolutional layers corresponding to industrial and household scenes respectively, and dynamic scene feature selection is achieved through the Gumbel-softmax probability distribution sampling method. During training, the batch size is set to 64, the AdamW optimizer is used, and the training is divided into two phases: the first 20 rounds use unlabeled industrial and household data mixed in a 7:3 ratio, with a fixed learning rate of 1 x 10^-4; the latter 30 rounds are trained separately for each scene, with 15 rounds each for industrial and household data, and the learning rate is reduced to 5 x 10^-5. A contrastive loss function is used for training loss, supplemented by data augmentation techniques such as random flipping, brightness adjustment, and Gaussian blur. After training, the model's feature extraction accuracy for industrial parts blocks improves to 88%, for household objects to 90%, and the feature selection accuracy of the scene adaptation layer reaches 92%.

[0045] Self-supervised learning models include a feature extraction backbone network and a scene adaptation layer. This enables self-supervised learning models to synergistically utilize the information carried by unsupervised and supervised signals, and simultaneously establish a perception of local details and a global semantic understanding of the scene. This provides robust feature representations for downstream multi-task learning that contain rich spatial details and have high-level semantic consistency, significantly enhancing the model's adaptability and discriminative ability when facing complex or unfamiliar environments.

[0046] Step S204: Input the standardized image patch dataset into the self-supervised learning model, and use the self-supervised learning model to extract second visual features from the labeled visual data.

[0047] In some embodiments, the self-supervised learning model extracts deep features from the labeled visual data through processing methods such as forward propagation, and the output obtained is the second visual feature. The second visual feature specifically refers to the deep semantic representation that is strongly related to the specific perceptual task and is extracted by the model from samples with explicit supervised information.

[0048] Step S205: Based on the first visual feature and the second visual feature, train a multi-task learning model so that the humanoid robot can perform at least two different visual perception tasks at the same time, and generate and execute corresponding action instructions according to the perception results of the multi-task learning model.

[0049] In some embodiments, the multi-task learning model includes at least two parallel task branches. The first task branch in the multi-task learning model receives block-encoded features and outputs a first-class perception result, which is a pixel-level segmentation result. The second task branch in the multi-task learning model receives class-encoded features and outputs a second-class perception result, which is a classification result or a regression estimation result.

[0050] In the architecture of the multi-task learning model, the first task branch receives block-encoded features containing rich spatial details and processes them through a series of pixel-level prediction techniques such as convolution and upsampling, ultimately outputting the first type of perception result, which is a segmentation map that classifies each pixel in the image. Simultaneously, the second task branch receives class-encoded features that characterize the overall content of the image and performs high-level semantic parsing using structures such as fully connected layers, thereby outputting the second type of perception result. This result can take the form of a classification label describing the category of the entire image or region, or a regression estimate that numerically predicts the target's location, pose, etc.

[0051] During the model training phase, multi-task sub-models for industrial and household scenarios were constructed and trained based on shared visual features extracted from the self-supervised learning model. For the industrial scenario, the defect segmentation branch receives block-encoded features, decodes them through three convolutional layers, and outputs a binary segmentation mask via a Softmax function; the pose estimation branch utilizes class-encoded features and predicts the pose vector through a two-layer fully connected network. Training employed a dynamically weighted loss function for one hundred iterations. As training progressed, the Dice coefficient for defect segmentation stabilized at 0.92, and the L1 loss for pose estimation stabilized at 0.3 mm. For the household scenario, the object segmentation branch outputs a multi-class mask with a similar structure, while the hazard classification branch outputs a behavior probability vector. Its training also employed a dynamic loss strategy for one hundred iterations, ultimately stabilizing the mIOU for object segmentation at 0.91 and the accuracy for hazard classification at 0.95. This training scheme enables two sub-models to efficiently and in parallel complete the core perception tasks in their respective scenarios based on a unified feature input. The collaborative mechanism allows the multi-task learning model to efficiently and synchronously process pixel-level segmentation and image-level recognition or estimation—two different granularities of visual tasks—based on the same set of shared underlying features. This not only significantly improves the computational efficiency and inference speed of the multi-task learning model but also promotes knowledge complementarity and collaborative optimization between different tasks through the feature sharing mechanism, thereby enhancing the overall performance and robustness of the multi-task learning model in completing comprehensive perception in complex scenarios.

[0052] In some embodiments, a dynamic loss function can be defined for the multi-task learning model. The dynamic loss function is used to jointly optimize at least two task branches. The dynamic loss function includes at least a first loss term and a second loss term. The first loss term corresponds to a first type of perceptual task, and the second loss term corresponds to a second type of perceptual task. During training, the weight coefficients of the first loss term and the second loss term in the total loss are dynamically adjusted based on the first visual feature and the second visual feature to obtain the trained multi-task learning model.

[0053] The core function of this dynamic loss function lies in its role in at least two task branches within the joint optimization model. These branches must contain at least a first loss term corresponding to the first type of perception task and a second loss term corresponding to the second type of perception task. The dynamic loss function typically employs a weighting strategy based on task uncertainty or gradient statistics, such as the task uncertainty weighting method. This method automatically balances the importance of different task losses using learnable noise parameters. Electronic devices can also adjust the weighting coefficients in real-time during training based on the relative magnitude of the task loss values ​​or the gradient magnitude. Specifically, during training, the multi-task learning model comprehensively considers the task relevance and learning state reflected in the first and second visual features learned from the data, and dynamically adjusts the weight ratio of the first and second loss terms in the total loss accordingly.

[0054] By adaptively balancing the learning progress of multiple tasks, the model is prevented from dominating the entire optimization process while other tasks fail to converge. This encourages the model to learn all related tasks more evenly. The resulting multi-task learning model not only performs stably on each individual task, but also achieves better synergy and overall generalization performance between tasks.

[0055] As an optional implementation, the trained multi-task learning model is evaluated using preset evaluation metrics. If the multi-task learning model meets the preset evaluation conditions, the trained multi-task learning model is obtained. The evaluation metrics include at least the mean accuracy for evaluating the segmentation task, the area under the receiver operating characteristic (ROC) curve for evaluating the classification task, and the error value for evaluating the regression task. Through multi-dimensional and multi-task quantitative standards, not only can the model's performance on each independent sub-task be objectively and comprehensively reflected, but it can also ensure that the model achieves overall performance balance and reliability when working collaboratively across tasks. This provides crucial quality verification and performance assurance for the actual deployment and application of the model in complex scenarios.

[0056] In some embodiments, if the perception result is determined to meet a first preset condition, a first type of control instruction is generated to control the humanoid robot to perform a first type of action; if the perception result meets a second preset condition, a second type of control instruction is generated to control the humanoid robot to perform a second type of action; wherein the first type of action is different from the second type of action.

[0057] When the perception results output by the humanoid robot's multi-task learning model meet the first preset conditions—for example, identifying a defective part in an industrial setting or detecting an elderly person in a normal activity state in a home setting—the electronic device automatically generates a corresponding first-type control command. This command is parsed into specific motion parameters by the robotic arm control SDK or gait planning module, thereby driving the robot to perform a first-type action, such as continuing assembly or conducting routine inspections. Conversely, if the perception results meet the second preset conditions, such as detecting a serious defect or identifying a dangerous behavior like a fall, the electronic device generates different second-type control commands, triggering a second-type action, such as pausing the assembly process and issuing an alarm or quickly approaching and providing assistance. By mapping the perception results into specific control commands through preset logical rules or lightweight decision-making models, a closed loop from environmental perception to action execution is achieved, significantly enhancing the humanoid robot's autonomous decision-making and real-time response capabilities. This allows it to autonomously switch work modes based on complex scene understanding without human intervention, greatly improving task execution efficiency and safety, and providing crucial technical support for completing diverse tasks in dynamic and changing environments.

[0058] During the model evaluation and implementation phase, for industrial scenarios, two thousand image patches randomly selected from labeled industrial data were used as the test set for evaluation. The results showed an average defect detection accuracy of 0.93, with 0.95 for cracks and 0.91 for scratches. Assembly pose errors on the X, Y, and Z axes were 0.42 mm, 0.38 mm, and 0.45 mm, respectively, and the rotation angle error was 0.25 degrees. All errors met industrial standard requirements. For home scenarios, one thousand five hundred image patches randomly selected from labeled home data were used for testing. The object recognition accuracy reached 0.96%, the area under the receiver operating characteristic (AUC) curve for hazardous behavior classification was 0.94, and the warning response time was 0.28 seconds. All performance indicators met the application specifications.

[0059] In the industrial setting of automotive bearing assembly, the Lidian F1 robot uses a camera mounted on its head to capture real-time images of the bearings at 1080P resolution and 120 frames per second. Each frame is preprocessed by normalization and 64x64 image block segmentation before being input into a multi-task learning model. The multi-task learning model simultaneously outputs a defect segmentation mask and assembly pose estimation results. If the mask detects a crack defect with a pixel percentage exceeding 5%, the execution module will trigger an emergency stop of the robotic arm within 45 milliseconds and mark the defect location on an industrial screen. Simultaneously, if the estimated assembly pose deviates from the standard pose by more than 0.5 millimeters, the robotic arm will automatically adjust its pose with an accuracy of ±0.1 millimeters based on the pose data output by the multi-task learning model, resulting in a 20% increase in assembly efficiency and a significant reduction in the defect miss rate from 5% to 0.8%.

[0060] In home-based elderly care scenarios, the Lidian D1 robot continuously captures visual data of the living room environment through a chest-mounted camera with a 120-degree wide-angle field of view and automatic switching to infrared mode at night. Each frame of the acquired image is pre-processed and then input into a multi-task learning model for analysis in real time. The multi-task learning model simultaneously outputs object segmentation masks and behavior classification results: when the segmentation mask identifies an overlap of more than 10% between the knife area and the child seat area, the humanoid robot will move to the target location at a speed of 0.3 meters per second and simultaneously issue a voice warning, "Please note that the knife is within the reach of a child." If the behavior classification result determines that the elderly person has fallen with a high probability of 0.98%, the execution module will immediately trigger a multi-level response mechanism, including voice inquiry about the situation, sending a warning SMS containing the time and address to a preset family member's mobile phone, and automatically calling the emergency center and sending precise location information if no response is received within ten seconds. This significantly reduces the rescue response time after an elderly person falls from the 30 seconds required by humans to the 15 seconds of automatic robot response, while achieving a 95% accuracy rate in warning of children coming into contact with dangerous objects.

[0061] Figure 3 is a schematic diagram of the visual perception architecture of a humanoid robot in one embodiment. As shown in Figure 3, visual data is collected through industrial camera A and home camera B, and the collected data is transmitted to the visual data collection module. At the same time, the data storage unit (2T server) stores the relevant data in TFRecord format. Then, the visual data is transmitted to the data preprocessing module for preprocessing, and then sequentially enters the model building module to build the model, the self-supervised model training module to carry out self-supervised training, and the multi-task model training module to carry out multi-task training. The trained model enters the model evaluation module for evaluation. Meanwhile, components such as industrial robotic arm C, home voice module D, mobile module E (wheeled unit), and emergency interruption unit are connected to the execution control module. Information such as the model evaluation results are also interacted with the execution control module. Finally, the execution control module performs execution control on industrial robotic arm C, home voice module D, mobile module E (wheeled unit), etc., and the emergency interruption unit can play a role when necessary.

[0062] Steps S201 to S205 of this embodiment involve acquiring unlabeled and labeled visual data of the target scene; preprocessing the unlabeled and labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; inputting the standardized image patch dataset into a self-supervised learning model, training the unlabeled visual data based on the self-supervised learning model to extract the first visual features of the target image patch data; inputting the standardized image patch dataset into the self-supervised learning model, using the self-supervised learning model to extract the second visual features of the labeled visual data; and training a multi-task learning model based on the first and second visual features to enable the humanoid robot to simultaneously perform at least two different visual perception tasks, and generating and executing corresponding action instructions based on the perception results of the multi-task learning model. Self-supervised learning reduces the dependence on a large amount of labeled data, and multi-task learning improves the model's generalization ability and efficiency in cross-task scenarios, while promoting the tight integration of perception and execution, thus enhancing the adaptability and operational consistency of the humanoid robot in cross-task environments.

[0063] Referring to Figure 4, this application embodiment also provides a visual perception device for a humanoid robot, which can implement the above-described method. The device includes: a data acquisition module 401, used to acquire unlabeled visual data and labeled visual data of a target scene; a data processing module 402, used to preprocess the unlabeled visual data and labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; a first extraction module 403, used to input the standardized image patch dataset into a self-supervised learning model, and train the unlabeled visual data based on the self-supervised learning model to extract the first visual features of the target image patch data; a second extraction module 404, used to input the standardized image patch dataset into the self-supervised learning model, and use the self-supervised learning model to extract the second visual features from the labeled visual data; and a model training module 405, used to train a multi-task learning model based on the first and second visual features, so that the humanoid robot can simultaneously perform at least two different visual perception tasks, and generate and execute corresponding action instructions according to the perception results of the multi-task learning model.

[0064] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0065] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0066] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0067] Please refer to Figure 5, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 501, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 502, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501. The input / output interface 503 is used to implement information input and output. The communication interface 504 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (such as USB, network cable, etc.) or by wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 505 transmits information between the various components of the device (such as processor 501, memory 502, input / output interface 503 and communication interface 504). The processor 501, memory 502, input / output interface 503 and communication interface 504 realize communication connection between each other within the device through the bus 505.

[0068] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0069] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0070] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0071] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0072] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0073] The visual perception method, apparatus, electronic device, storage medium, and program product for humanoid robots provided in this application acquire unlabeled and labeled visual data of a target scene; preprocess the unlabeled and labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; input the standardized image patch dataset into a self-supervised learning model, and train the unlabeled visual data based on the self-supervised learning model to extract the first visual features of the target image patch data; input the standardized image patch dataset into the self-supervised learning model, and use the self-supervised learning model to extract the second visual features of the labeled visual data; based on the first and second visual features, train a multi-task learning model so that the humanoid robot can simultaneously perform at least two different visual perception tasks, and generate and execute corresponding action instructions according to the perception results of the multi-task learning model, thereby enhancing the adaptability and operational coherence of the humanoid robot in cross-task environments.

[0074] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0075] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0078] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0081] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A visual perception method for a humanoid robot, characterized in that, The method includes the following steps: acquiring unlabeled visual data and labeled visual data of a target scene; preprocessing the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; inputting the standardized image patch dataset into a self-supervised learning model, processing the unlabeled visual data based on the self-supervised learning model to extract a first visual feature of the target image patch data; inputting the standardized image patch dataset into the self-supervised learning model, using the self-supervised learning model to extract a second visual feature from the labeled visual data; training a multi-task learning model based on the first visual feature and the second visual feature, so that the humanoid robot can simultaneously perform at least two different visual perception tasks, and generate and execute corresponding action instructions according to the perception results of the multi-task learning model.

2. The method according to claim 1, characterized in that, The preprocessing of the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset includes: normalizing the images in the unlabeled visual data and the labeled visual data to a preset size and resolution; dividing the normalized images into non-overlapping image patches, and for the image patches corresponding to the labeled visual data, calculating the proportion of the target region in the image patch based on its annotation information; and selecting image patches whose target region proportion exceeds a preset threshold to form the standardized image patch dataset.

3. The method according to claim 1, characterized in that, The self-supervised learning model includes a feature extraction backbone network and a scene adaptation layer; the feature extraction backbone network is used to extract the first visual feature and the second visual feature from the target image patch data; The scene adaptation layer is used to process the first visual feature and the second visual feature to output block-coded features containing spatial location information and class-coded features containing global semantic information.

4. The method according to claim 3, characterized in that, The multi-task learning model includes at least two parallel task branches; Before generating and executing the corresponding action instruction based on the perception result of the multi-task learning model, the method further includes: receiving the block-encoded features through a first task branch in the multi-task learning model and outputting a first type of perception result; the first type of perception result is a pixel-level segmentation result; receiving the class-encoded features through a second task branch in the multi-task learning model and outputting a second type of perception result; the second type of perception result is a classification result or a regression estimation result.

5. The method according to claim 4, characterized in that, The step of training a multi-task learning model based on the first visual feature and the second visual feature to enable a humanoid robot to simultaneously perform at least two different visual perception tasks includes: defining a dynamic loss function for the multi-task learning model; the dynamic loss function is used to jointly optimize the at least two task branches, and the dynamic loss function includes at least a first loss term and a second loss term, where the first loss term corresponds to the first type of perception task and the second loss term corresponds to the second type of perception task; during training, the weight coefficients of the first loss term and the second loss term in the total loss are dynamically adjusted based on the first visual feature and the second visual feature to obtain the trained multi-task learning model.

6. The method according to claim 1, characterized in that, After training a multi-task learning model based on the first visual feature and the second visual feature to enable the humanoid robot to perform at least two different visual perception tasks simultaneously, the method further includes: evaluating the trained multi-task learning model using a preset evaluation metric; if the multi-task learning model is determined to meet the preset evaluation conditions, then a trained multi-task learning model is obtained; the evaluation metric includes at least the mean average accuracy used to evaluate the segmentation task, the area under the receiver operating characteristic curve used to evaluate the classification task, and the error value used to evaluate the regression task.

7. The method according to claim 1, characterized in that, The step of generating and executing corresponding action instructions based on the perception results of the multi-task learning model includes: if it is determined that the perception results meet a first preset condition, then generating a first type of control instruction to control the humanoid robot to perform a first type of action; if the perception results meet a second preset condition, then generating a second type of control instruction to control the humanoid robot to perform a second type of action; wherein the first type of action is different from the second type of action.

8. A visual perception device for a humanoid robot, characterized in that, The device includes: a data acquisition module for acquiring unlabeled visual data and labeled visual data of a target scene; a data processing module for preprocessing the unlabeled visual data and the labeled visual data to obtain a standardized image patch dataset; the standardized image patch dataset contains multiple target image patch data; a first extraction module for inputting the standardized image patch dataset into a self-supervised learning model, training the unlabeled visual data based on the self-supervised learning model, and extracting a first visual feature from the target image patch data; a second extraction module for inputting the standardized image patch dataset into the self-supervised learning model, and extracting a second visual feature from the labeled visual data using the self-supervised learning model; and a model training module for training a multi-task learning model based on the first visual feature and the second visual feature, so that the humanoid robot can simultaneously perform at least two different visual perception tasks, and generate and execute corresponding action instructions according to the perception results of the multi-task learning model.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images