Scene backtracking method, system, equipment and product of humanoid robot

Abstract

The invention discloses a scene backtracking method, system, equipment and product for a humanoid robot, and the method comprises the steps: triggering the humanoid robot to carry out multi-source data collection processing according to an operation instruction, carrying out the control processing of a data collection process through the operation state of the humanoid robot, and obtaining a multi-source data set; the multi-source data set comprises positioning data, environment video data and target object state data; performing data preprocessing and sample extraction processing on the environment video data according to the positioning data to obtain a sample image; performing target feature recognition and image screening processing on the sample image to obtain a scene image; and performing visual backtracking processing on the positioning data, the scene image and the target object state data to obtain a scene backtracking result. The working efficiency of the humanoid robot can be improved, and the humanoid robot can be widely applied to the technical field of robots.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, system, device and product for scene backtracking of a humanoid robot. Background Technology

[0002] In related technologies, there is a method where robots use a single laser for simultaneous localization and mapping (SLAM), meaning the robot relies on lasers for navigation and positioning. However, in practical applications, it has been found that single-laser positioning is easily affected by environmental interference, making it impossible to utilize the advantage of centimeter-level indoor positioning for accurate backtracking, thus affecting the working efficiency of humanoid robots.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a scene backtracking method, system, device, and product for humanoid robots, which can improve the working efficiency of humanoid robots.

[0005] To achieve the above objectives, one aspect of this application proposes a scene backtracking method for a humanoid robot, the method comprising: The humanoid robot is triggered to perform multi-source data acquisition and processing according to the work instructions, and the data acquisition process is controlled and processed by the work status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data; Based on the positioning data, the environmental video data is preprocessed and sample extraction is performed to obtain sample images; The sample images are subjected to target feature recognition and image filtering processing to obtain scene images; The location data, the scene image, and the target object state data are subjected to visualization backtracking processing to obtain the scene backtracking result.

[0006] In some embodiments, controlling the data acquisition process based on the working state of the humanoid robot to obtain a multi-source dataset includes: Sensor data is acquired through multi-source sensors; The task instruction is decomposed and progress is identified to obtain the execution progress of the task instruction; Based on the sensor data and the progress of the execution of the work instructions, the humanoid robot is subjected to probability prediction processing of its work state to obtain the work state; According to the operation status, the humanoid robot is controlled to perform anomaly detection and data acquisition and processing. The positioning data is obtained by data fusion through the synchronous positioning and map building module, the visual navigation module and the semantic map, and the environmental video data and the target object status data are collected simultaneously.

[0007] In some embodiments, the location data obtained by data fusion through a synchronous positioning and map building module, a visual navigation module, and a semantic map includes: The environmental map is obtained by performing environmental map construction and dynamic obstacle filtering through the synchronous positioning and map building module. The visual navigation module collects environmental feature points and performs trajectory correction processing on the environmental map based on the environmental feature points to obtain trajectory data. The location data is obtained by performing positioning calibration processing on the trajectory data using the landmark data of the semantic map.

[0008] In some embodiments, the step of performing data preprocessing and sample extraction processing on the environmental video data based on the positioning data to obtain sample images includes: Based on the positioning data, the humanoid robot's movement speed is matched and time is recorded to obtain the data clipping start timestamp. Based on the data cropping start timestamp, redundant segments are removed from the environmental video data to obtain the target video data. The target video data is processed by image frame extraction according to a preset sampling frequency to obtain the sample image.

[0009] In some embodiments, the step of performing target feature recognition and image filtering processing on the sample image to obtain a scene image includes: The sample image is subjected to target feature recognition processing to obtain target features; The target features are input into a convolutional neural network for confidence assessment to obtain the target confidence score. The sample images are filtered based on the target confidence level to obtain the scene image.

[0010] In some embodiments, the visualization backtracking processing of the positioning data, the scene image, and the target object state data to obtain the scene backtracking result includes: The positioning data, the scene image, and the target object state data are correlated and mapped to obtain mapped data. The mapped data is subjected to fault feature extraction and fusion processing to obtain fault features; The fault characteristics are subjected to fault prediction and root cause analysis to obtain the scenario backtracking results.

[0011] In some embodiments, the step of performing association mapping processing on the positioning data, the scene image, and the target object state data to obtain mapping data includes: The location data, the scene image, and the target object status data are time-stamped and spatially bound to obtain bound data. The bound data is subjected to trajectory index generation and matching processing to obtain matching data; A structured association table is constructed based on the matching data, and anomaly compensation processing is performed on the structured association table to obtain the mapping data.

[0012] To achieve the above objectives, another aspect of this application proposes a scene retracing system for a humanoid robot, the system comprising: The data acquisition module is used to trigger the humanoid robot to perform multi-source data acquisition and processing according to the operation instructions, and to control the data acquisition process through the operation status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data; The sample extraction module is used to perform data preprocessing and sample extraction processing on the environmental video data based on the positioning data to obtain sample images; The image filtering module is used to perform target feature recognition and image filtering processing on the sample images to obtain scene images; The visualization backtracking module is used to perform visualization backtracking processing on the positioning data, the scene image, and the target object status data to obtain the scene backtracking result.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above. The embodiments of this application include at least the following beneficial effects: This application provides a scene backtracking method, system, device, and product for a humanoid robot. This solution triggers the humanoid robot to perform multi-source data acquisition and processing based on work instructions, and controls the data acquisition process through the humanoid robot's work status to obtain a multi-source dataset. The multi-source dataset includes positioning data, environmental video data, and target object status data. This application embodiment controls the data acquisition process by detecting the humanoid robot's work status, enabling on-demand data acquisition and improving data acquisition efficiency. Furthermore, this solution also performs visual backtracking processing on the positioning data, scene images, and target object status data to obtain scene backtracking results. This not only enables basic data display but also allows for scene-based analysis, improving the working efficiency of the humanoid robot. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of a scene backtracking method for a humanoid robot provided in an embodiment of this application; Figure 3 This is a schematic diagram of an indoor positioning fusion strategy provided in an embodiment of this application; Figure 4 This is a sampling frequency calculation logic diagram provided in an embodiment of this application; Figure 5 This is a flowchart of a location anomaly handling method provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a scene tracing system for a humanoid robot provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1) Humanoid robots, also known as bionic robots, are robots designed to mimic human appearance and behavior, especially those with similar physiques to humans. The structural design of humanoid robots represents a remarkable reshaping of the human body, requiring not only interdisciplinary integration but also the culmination of cutting-edge technologies. Their design principles primarily include the following aspects: the organic integration of bionics and mechanical engineering, breakthroughs in the integration of sensing technology and control theory, and precise coordination between drive mechanisms and execution actions.

[0023] 2) Simultaneous Localization and Mapping (SLAM) is a core technology that enables robots to locate themselves in unknown environments and build environmental maps in real time using sensors. Its core principle is to fuse measurement data from sensors (such as LiDAR, cameras, and millimeter-wave radar) to simultaneously estimate the robot's own trajectory and the coordinates of environmental features. This technology is applied in autonomous vehicles, drones, and autonomous underwater robots, and extends to scenarios such as spatial positioning for AR / VR devices, underground parking lot navigation, and intelligent mobility aids. SLAM collects environmental information (such as distance, angle, and image features) through sensors, combines it with motion control data, and simultaneously calculates the robot's pose and updates the environmental map.

[0024] As humanoid robots penetrate complex indoor environments (such as factory workshops and shopping malls), the industry consensus on positioning and navigation technology has emerged: "Indoor navigation relies on SLAM, visual navigation, and semantic maps, while large outdoor venues use BeiDou or GPS as supplementary technologies." Among related technologies, SLAM-based robot navigation and positioning methods often employ single-laser SLAM positioning. After long-term operation, changes in environmental characteristics (such as equipment movement in workshops) cause map drift, degrading positioning accuracy from an initial ±2cm to ±10cm, failing to meet the centimeter-level requirements of equipment inspection. Visual navigation service robots are affected by lighting conditions, significantly reducing positioning success rates and exhibiting insufficient positioning accuracy and stability. Furthermore, most robots use a "continuous data acquisition" mode, resulting in a large total data volume and high storage costs due to the combination of SLAM map data and video data. When equipment malfunctions or service complaints arise, it is impossible to revisit and reproduce the operational process, leading to low troubleshooting efficiency and issues of data redundancy and weak backtracking capabilities.

[0025] In view of this, this application provides a method, system, device, and product for scene backtracking of humanoid robots. This solution relates to the field of humanoid robot technology, specifically to a method, robot, and system for autonomous operation and scene backtracking of humanoid robots in indoor scenarios (industrial workshop equipment inspection, shopping mall guidance, hotel services, relying on SLAM, visual navigation, and semantic mapping to achieve centimeter-level positioning) and large outdoor venues (open-air factories, commercial squares, using Beidou / GPS positioning as an auxiliary method). This application constructs a technical framework of precise positioning, dynamic acquisition, intelligent filtering, and associated backtracking: solving the indoor positioning stability problem through multi-technology fusion, reducing redundancy through scenario-triggered acquisition, judging the operation status through fuzzy logic, and finally achieving high-precision backtracking through three-dimensional association of "SLAM trajectory - key image - status data". This can solve the problems of insufficient indoor positioning accuracy (such as SLAM mapping drift), lag when switching between indoor and outdoor positioning, large data redundancy, and difficulty in tracing operation anomalies in related technologies, thereby improving the intelligence and manageability of robot operations in real-world scenarios.

[0026] Figure 1 This is a schematic diagram illustrating the implementation environment of a method provided in an embodiment of this application. (Refer to...) Figure 1 The main hardware and software components of this implementation environment include a humanoid robot 101 and a server 102, with the humanoid robot 101 and server 102 communicating with each other. The method can be executed based on the interaction between the humanoid robot 101 and server 102. Furthermore, the humanoid robot 101 and server 102 can be nodes in a blockchain, but this embodiment does not specifically limit this.

[0027] Figure 2 This is an optional flowchart of a scene backtracking method for a humanoid robot provided in an embodiment of this application. Figure 2 The method may include, but is not limited to, steps S102 to S204.

[0028] Step S201: Trigger the humanoid robot to perform multi-source data acquisition and processing according to the work instruction, and control the data acquisition process through the work status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data; Step S202: Based on the positioning data, perform data preprocessing and sample extraction processing on the environmental video data to obtain sample images; Step S203: Perform target feature recognition and image filtering processing on the sample image to obtain a scene image; Step S204: Perform visualization backtracking processing on the positioning data, the scene image, and the target object state data to obtain the scene backtracking result.

[0029] Steps S201 to S204 of this embodiment, as shown in the present application, trigger the humanoid robot to perform multi-source data acquisition and processing according to the work instruction. Specifically, this embodiment can activate the multi-source sensor module to acquire multi-source data when the humanoid robot receives the work instruction or detects that the movement distance within the work area exceeds the preset distance threshold for scene adaptation (0.1m in industrial scenes / 0.3m in commercial scenes). In a feasible embodiment, this embodiment can acquire centimeter-level precision work path positioning data (including coordinates and attitude angles) through a SLAM (Simultaneous Localization and Mapping) module, a visual navigation module (binocular camera), and semantic map matching in indoor scenes, and simultaneously acquire environmental video data and target object status data (equipment operating parameters in industrial scenes / interaction data of service objects in commercial scenes); in large outdoor venues, a Beidou / GPS positioning module is used to assist in positioning and data acquisition, thereby acquiring multi-source data. This embodiment also establishes a work status probability prediction model based on fuzzy logic, outputs the work completion probability, and stops acquisition after the probability is reached. This application embodiment also calculates a preset sampling frequency based on robot movement speed, task priority coefficient, etc., extracts sample images, and filters key images according to a scenario-based confidence threshold to obtain scene images. Finally, when connected to the network, the data is encrypted and uploaded to the cloud. During terminal backtracking, the task process is reproduced through "trajectory-image-state" three-dimensional data visualization to obtain scene backtracking results. This invention embodiment can adapt to actual application scenarios of indoor and outdoor positioning technology, reduces more than 85% of redundant data through accurate collection and filtering, takes into account both industrial fault tracing and commercial service optimization, and improves the intelligence and traceability of robot operations.

[0030] In step S201 of some embodiments, the humanoid robot is triggered to perform multi-source data acquisition and processing according to the operation instruction, and the data acquisition process is controlled and processed by the operation status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data.

[0031] Specifically, in this embodiment of the application, when the humanoid robot receives an external operation instruction or detects through the positioning module that its own movement distance in the operation area exceeds a preset distance threshold, the multi-source sensor module is activated to synchronously collect a multi-source dataset, which includes positioning data, environmental video data, and target object status data.

[0032] For example, positioning data can be generated in indoor scenes using a SLAM module (laser SLAM or visual SLAM), a visual navigation module (binocular camera + visual odometry), and semantic map matching to output real-time coordinates (x, y, z) and attitude angles (roll α, pitch β, yaw γ) with centimeter-level accuracy; in outdoor large-scale venues (such as open-air factories or large commercial plazas), a BeiDou / GPS positioning module + IMU (Inertial Measurement Unit) can be used to output sub-meter-level positioning data. Environmental video data is a real-time video stream of the work area. Target object status data includes equipment operating parameters (temperature, vibration frequency, voltage, etc.) in industrial scenarios and service object interaction behavior data (interaction duration, guidance path deviation, voice command recognition rate, etc.) in commercial scenarios. It is important to note the preset distance threshold. The robot's operational precision level can be set, for example, in industrial scenarios (such as equipment inspection). =0.1m (suitable for indoor centimeter-level positioning accuracy), for commercial scenarios (such as shopping guides). =0.3m (suitable for indoor centimeter-level positioning accuracy).

[0033] In some feasible embodiments, the process of controlling the data acquisition process through the working state of the humanoid robot to obtain a multi-source dataset includes: Sensor data is acquired through multi-source sensors; The task instruction is decomposed and progress is identified to obtain the execution progress of the task instruction; Based on the sensor data and the progress of the execution of the work instructions, the humanoid robot is subjected to probability prediction processing of its work state to obtain the work state; According to the operation status, the humanoid robot is controlled to perform anomaly detection and data acquisition and processing. The positioning data is obtained by data fusion through the synchronous positioning and map building module, the visual navigation module and the semantic map, and the environmental video data and the target object status data are collected simultaneously.

[0034] In this embodiment, sensor data is acquired through multi-source sensors. This data can be directly collected by force sensors (such as strain gauge force sensors) built into the robot's end effector, providing real-time feedback on the load magnitude. The distance to environmental obstacles is also obtained through the depth detection function of LiDAR or a binocular camera. LiDAR calculates the distance to obstacles using the "time-of-flight method" (measuring the time it takes for a laser pulse to travel from emission to reflection); the binocular camera calculates depth information using the "parallax principle" (utilizing the pixel offset between the left and right camera images), ultimately outputting the real-time distance between the obstacle and the robot. In industrial scenarios, real-time equipment parameters are collected using industrial sensors (such as temperature sensors, vibration sensors, and voltage sensors). These parameters are compared with pre-stored "equipment standard parameters" (from equipment manuals or historical normal operation data, such as rated temperature, vibration frequency threshold, and rated voltage), and the deviation percentage is calculated as "(measured value - standard value) / standard value × 100%". In commercial scenarios, the actual interaction time can be recorded by the timing module and compared with the system's "preset expected interaction time" (set according to historical service data or task requirements, such as "the expected time to guide to the store is 60 seconds") to calculate "(actual time - expected time) / expected time × 100%" and obtain the deviation percentage.

[0035] Specifically, this embodiment of the application will further perform task decomposition and progress identification processing on the work instructions to obtain the execution progress of the work instructions. The robot breaks down the work instructions into several sub-tasks (e.g., "inspect 10 pieces of equipment" is broken down into 10 sub-tasks, and each time one sub-task is completed, ...). (Increase by 10%); at the same time, combining the action completion signals of the end effector (such as feedback of clamping / releasing actions) and the recognition results of the task target by the vision sensor (such as the recognition success signal of the equipment nameplate / store LOGO), the completion ratio of the sub-task is statistically analyzed in real time, and finally a continuous progress value of 0-100% is obtained.

[0036] For example, in this embodiment of the application, the robot performs work state probability prediction processing based on sensor data and the progress of work instruction execution, and outputs the work completion probability. The goal is to dynamically determine the robot's current work status (completed / in progress / needs to be paused), thereby achieving "on-demand data collection." (Complete the task): Stop data collection to avoid generating redundant data; if (Work in progress): Continuously collect data to support the recording of the work process; if (Work needs to be suspended): Trigger exception handling to ensure work safety and task continuity.

[0037] When the fuzzy logic model outputs Corresponding to "Pause Operation" (such as matching rule "") or or or The robot performs the following operations: Step 1: Immediately pause the current action; The end effector stops its operation (such as the detection action during inspection or the guidance action during sales), and the mobile chassis pauses path planning to avoid malfunctions or service defects caused by continuing to operate under abnormal conditions.

[0038] Step 2: Anomaly cause detection and feedback; Multi-source sensors focus on collecting data related to the "reason for suspension" (such as obstacle distribution, load change trend, and abnormal parameters of the target object), and send "operation suspension alarms" to the cloud / terminal, along with preliminary cause analysis (such as "suspension due to obstacle being too close" or "suspension due to excessive equipment temperature deviation").

[0039] Step 3: Implement the preset response strategy; Different recovery logic is triggered depending on the reason for the pause: If due to "environmental obstacles being too close" ": Initiate obstacle avoidance algorithm and replan path; pending..." And recalculate (When 40% is taken for industrial scenarios and 50% for commercial scenarios, operations will resume.)

[0040] If the problem is due to "excessive end load" ": Pause the operation and send an "Overload Alarm". You can choose to "Wait for manual load adjustment" or "Wait for the load to decrease naturally within a short period within the safety threshold". Then, operations resumed.

[0041] If the target object's state deviation is too large ": Industrial scenario: Trigger "Equipment Anomaly Warning", record the current abnormal data and suspend inspection, waiting for maintenance personnel to confirm / handle; Commercial applications: Adjust interaction strategies (such as extending interaction duration, reinterpreting service content), and update in real time. And recalculate ;when Service will be restored in time.

[0042] Step 4: Continuous monitoring and reassessment; During the pause, the multi-source sensors re-collect data every second and input it into the fuzzy logic model for continuous calculation. If the pause timeout exceeds 5 minutes in industrial scenarios and 2 minutes in commercial scenarios, a "pause timeout alert" will be sent to request manual intervention.

[0043] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application control the data collection process through the working status of the humanoid robot, which can dynamically collect and filter the amount of data, reduce cloud storage costs, and improve the efficiency of industrial fault diagnosis and commercial complaint handling, thereby further improving the data collection efficiency of the humanoid robot.

[0044] In some embodiments, the location data obtained by data fusion through a synchronous positioning and map building module, a visual navigation module, and a semantic map includes: The environmental map is obtained by performing environmental map construction and dynamic obstacle filtering through the synchronous positioning and map building module. The visual navigation module collects environmental feature points and performs trajectory correction processing on the environmental map based on the environmental feature points to obtain trajectory data. The location data is obtained by performing positioning calibration processing on the trajectory data using the landmark data of the semantic map.

[0045] In the embodiments of this application, please refer to Figure 3 This application embodiment constructs an environmental map in real time using laser SLAM (industrial) / visual SLAM (commercial). Specifically, it outputs the robot's relative coordinates and reduces mapping drift through a "dynamic obstacle filtering algorithm" (removing moving targets such as personnel and temporary materials), stabilizing the positioning accuracy within ±1cm. Then, it collects environmental feature points using a binocular camera, calculates visual odometry data, and performs trajectory correction processing on the environmental map based on these feature points, correcting the SLAM trajectory every 100ms to control the drift within ±3cm, thus obtaining trajectory data. Finally, by using pre-stored key landmarks in the work area, such as equipment nameplates in industrial scenarios and store logos in commercial scenarios, the robot automatically calibrates its positioning coordinates when it detects a landmark, improving accuracy to ±1 to 3cm, meeting the requirements for high-precision indoor operations.

[0046] For example, indoor positioning data adopts a fusion strategy of "SLAM + visual navigation + semantic map": the SLAM module builds an environmental map of the work area in real time and outputs the robot's relative map coordinates; the visual navigation module calculates visual odometry data by matching feature points of binocular cameras to correct SLAM mapping drift; the semantic map matches key landmarks (such as equipment nameplates in industrial scenarios and store logos in commercial scenarios) to further calibrate the positioning accuracy to the centimeter level (±1 to 3cm); in large outdoor venues, "BeiDou / GPS + IMU" fusion is adopted, and the positioning results are output through extended Kalman filtering (BeiDou / GPS weight 0.7, IMU weight 0.3).

[0047] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application obtain positioning data by using a data fusion scheme of SLAM + visual navigation + semantic map in indoor environment, which can stabilize the positioning accuracy at ±1 to 3cm, improve the anti-light interference capability (positioning success rate ≥90% in commercial scenarios) and the anti-map drift capability (drift ≤2cm in industrial scenarios 24 hours of operation); and the indoor-outdoor switching delay is <0.5 seconds, reducing lag.

[0048] In some embodiments, the step of performing data preprocessing and sample extraction processing on the environmental video data based on the positioning data to obtain sample images includes: Based on the positioning data, the humanoid robot's movement speed is matched and time is recorded to obtain the data clipping start timestamp. Based on the data cropping start timestamp, redundant segments are removed from the environmental video data to obtain the target video data. The target video data is processed by image frame extraction according to a preset sampling frequency to obtain the sample image.

[0049] In this embodiment, the humanoid robot's movement speed is matched based on positioning data to determine the timestamp for cropping the environmental video data, and the corresponding data cropping start timestamp is recorded. Then, the environmental video data is cropped based on the data cropping start timestamp, and the target video data is obtained by deleting redundant segments. Furthermore, this embodiment uses a preset sampling frequency to perform image frame extraction processing on the target video data to obtain sample images.

[0050] For example, by matching positioning data with robot movement speed (v, in m / s), when v < a preset speed threshold at the end of the work path ( When the speed is 0.2 m / s for industrial use and 0.5 m / s for commercial use, determine the starting timestamp (T) for data trimming, delete redundant segments after T in the environmental video data, and obtain the target video data; if the duration of the target video data exceeds the preset duration threshold ( (Industrial 5min, Commercial 3min), delete segments that exceed T at the beginning; please refer to Figure 4 Image frames are extracted from the target video data at a preset sampling frequency (F) to form sample images. The expression for the preset sampling frequency is as follows: ; Where v represents the robot's moving speed (calculated from the time difference between adjacent coordinates using SLAM / visual navigation and positioning data); ρ represents the task priority coefficient (task accuracy requirement α × task urgency β, for industrial inspection α=1.2, β=1.0, so ρ=1.2; for commercial sales guide α=1.0, β=0.8, so ρ=0.8); d represents the minimum sampling distance (0.2m for industrial, 0.6m for commercial); ε represents the environmental complexity coefficient (quantified by the number of obstacles within a unit area of ​​10m², 0-2 obstacles ε=1.0, 3-5 obstacles ε=0.8, 6-8 obstacles ε=0.6).

[0051] One of the above technical solutions has the following advantages or beneficial effects: the embodiments of this application can extract sample images, which provides a data basis for subsequent image screening and processing.

[0052] In some embodiments, the step of performing target feature recognition and image filtering processing on the sample image to obtain a scene image includes: The sample image is subjected to target feature recognition processing to obtain target features; The target features are input into a convolutional neural network for confidence assessment to obtain the target confidence score. The sample images are filtered based on the target confidence level to obtain the scene image.

[0053] Specifically, in this embodiment, a feature recognition model can be used to perform target feature recognition processing on sample images to extract target features. The extracted target features are then input into a convolutional neural network for confidence assessment to obtain the confidence level of the corresponding target. Based on a pre-set confidence threshold, the calculated target confidence level can be compared with the calculated confidence level, and the sample images can be filtered based on different comparison results to obtain the corresponding scene images.

[0054] In this embodiment, target feature recognition is performed on the sample images. In industrial scenarios, the target feature is equipment malfunction characteristics, and in commercial scenarios, the target feature is the interaction actions of service recipients. Then, a convolutional neural network (CNN) is used to output the confidence score (C, 0-100%) for each frame. When C ≥ a preset confidence threshold (…),… When the industrial sector accounts for 80% and the commercial sector for 70%, select the 1-3 frames with the highest confidence as key images; when all C < When selecting frames, the time interval is calculated as "total target video duration ÷ number of selected frames (3 frames for industrial use / 2 frames for commercial use)," prioritizing frames with robot posture angle changes ≥5° or ΔS ≥5%. This embodiment can also enable infrared thermal imaging to generate keyframes and mark them as "visual failures" in a completely dark environment, thereby obtaining the corresponding scene image.

[0055] One of the above technical solutions has the following advantages or beneficial effects: The scene images obtained by the embodiments of this application based on confidence level screening can improve the humanoid robot's ability to recognize scenes and provide a data foundation for subsequent scene backtracking processing.

[0056] In some embodiments, the visualization backtracking processing of the positioning data, the scene image, and the target object state data to obtain the scene backtracking result includes: The positioning data, the scene image, and the target object state data are correlated and mapped to obtain mapped data. The mapped data is subjected to fault feature extraction and fusion processing to obtain fault features; The fault characteristics are subjected to fault prediction and root cause analysis to obtain the scenario backtracking results.

[0057] Specifically, this application embodiment stores positioning data, scene images, and target object status data, and enables a backtracking function. Specifically, when the robot is communicating normally with the cloud server, a dynamic key is generated using "robot unique identifier + timestamp" to encrypt and upload positioning data, key images with watermarks (robot number + operation time), and status data. In case of communication interruption, data is temporarily stored on the local SSD and uploaded first upon recovery. When the terminal initiates a backtracking request, the server matches the operation time / area data and sends it to the terminal. The terminal generates the operation path trajectory (based on SLAM / visual navigation positioning data), a key image timeline, and a status data trend curve, enabling visualized backtracking.

[0058] In this embodiment, during visual backtracking, the terminal correlates and maps location data, scene images, and target object state data to obtain mapped data. Then, fault feature extraction and fusion processing are performed on the mapped data. A fault feature recognition model is built into the terminal. This model is based on deep learning, and its training set includes historical fault images and corresponding state data. Image feature extraction processing is performed on the backtracking data, specifically identifying visual features such as "abnormal textures (e.g., oil stains)" and "shape deviations (e.g., loose parts)" from key images. State data feature extraction is also performed, extracting numerical features such as "parameter mutations (e.g., sudden temperature rises)" and "continuous deviations from thresholds (e.g., long-term excessive vibration)" from trend curves. Then, multi-feature fusion processing is performed using a weighted fusion algorithm (visual feature weight 0.6, state data weight 0.4, adjustable according to the scenario) to output "equipment fault suspicion level". Finally, this embodiment performs fault prediction and root cause analysis. If the "equipment fault suspicion" is greater than or equal to a preset threshold (e.g., 80%), the terminal automatically triggers a "fault prediction alarm." Combined with a historical fault database (similar equipment fault cases stored in the cloud), a similarity matching algorithm (calculating the cosine similarity between current features and historical fault features) is used to push the "most likely fault type (e.g., bearing wear, seal aging)" and "historical maintenance plan reference." Simultaneously, the terminal generates a "fault evolution trajectory map": based on the time series of state data, it fits the parameter change trend to predict the time nodes of the fault's "emergence-development-outbreak," assisting maintenance personnel in early intervention.

[0059] For example, embodiments of this application can also decompose interactive scene elements during visual backtracking via the terminal. This involves decomposing SLAM / visual navigation path trajectory data (the customer's movement path following the robot), key images (customer interaction actions, such as pausing or gestures), and status data (interaction duration, voice command recognition rate, and path deviation) into three dimensions: "spatial path - behavioral actions - service parameters." Then, service anomaly pattern recognition is performed. The terminal uses behavior recognition and clustering algorithms to identify path anomalies in the backtracking data. By comparing the "planned path" with the "actual trajectory," and using a path deviation integral algorithm (calculating the area difference between the actual trajectory and the planned path), abnormal segments such as "detours" and "repeated paths" are located. Next, behavioral anomaly recognition processing is performed. By identifying actions such as "customer hesitation" and "repeatedly checking the environment" from key images, and combining this with the interaction duration curve, suspected scenarios of "unclear service guidance" are identified. Finally, multimodal anomaly fusion processing is performed. By spatiotemporally associating "path anomaly segments," "behavioral anomaly frames," and "service parameter anomalies (such as a sudden drop in voice recognition rate)," the core time periods and locations of "poor service experience" are marked. This application embodiment also generates corresponding service optimization suggestions. If a "service anomaly mode" is identified, the terminal generates optimization suggestions based on the service scenario rule base (pre-stored knowledge such as "shopping mall circulation design and customer psychology") as follows: Path level: recommend "simpler guidance paths" (such as new routes that avoid densely populated areas); Interaction level: suggest "optimizing the voice command library (supplementing standard answers to frequently asked questions)" and "adding visual cues at key locations (such as playing guidance voices at intersections)"; Decision support: push "heat maps of service anomaly hotspot areas" to operations personnel to assist in adjusting robot deployment or human service points.

[0060] One of the above technical solutions has the following advantages or beneficial effects: The visual backtracking provided by the embodiments of this application is not only a display of "trajectory + image + curve", but also realizes predictive judgment of industrial failures and optimized attribution of commercial services through multi-source data fusion and intelligent model analysis, so that the backtracking process is upgraded from "passive viewing" to "active decision support".

[0061] In some embodiments, the step of performing association mapping processing on the positioning data, the scene image, and the target object state data to obtain mapping data includes: The location data, the scene image, and the target object status data are time-stamped and spatially bound to obtain bound data. The bound data is subjected to trajectory index generation and matching processing to obtain matching data; A structured association table is constructed based on the matching data, and anomaly compensation processing is performed on the structured association table to obtain the mapping data.

[0062] In this embodiment of the application, by using centimeter-level positioning data (accurate to the three-dimensional coordinates of the device / component), key images (marking device fault characteristics, such as oil leakage, part deformation), and status data trend curves (time series of parameters such as temperature, vibration, and voltage) from SLAM / visual navigation to perform a three-dimensional association of "positioning-image-status" with time stamps and spatial coordinates, the status data and key images are anchored to the physical location of the specific device on the SLAM map on the terminal interface.

[0063] For example, in this embodiment of the application, the acquisition of three types of data during robot operation is synchronized with the spatial coordinates in real time via hardware clock, achieving initial binding: Timestamp synchronization: Utilizing the high-precision clock (millisecond-level synchronization) of the robot's main control module, a unified timestamp is applied to the SLAM trajectory data (map / pose output per frame), key images (acquisition per frame), and state data (sensor sampling), ensuring the initial alignment of the three types of data in the time dimension. Spatial coordinate binding: The SLAM module outputs the robot's three-dimensional absolute coordinates (x, y, z) and attitude angles in the environmental map in real time; when the vision camera acquires images, it synchronously records the absolute coordinates of the current SLAM output as the "acquisition position coordinates" of the image; when the state sensor (such as a temperature or vibration sensor, or a commercial interactive sensor) samples, it is also associated with the current SLAM absolute coordinates as the "state data acquisition position coordinates".

[0064] This application embodiment also uses algorithms to perform precise temporal and spatial matching of three types of data, establishing an association index centered on the "trajectory index ID": SLAM trajectory index generation: The pose sequence output by SLAM is sorted by timestamp, and a unique trajectory index ID is generated at fixed intervals (e.g., 500ms) or pose changes (e.g., movement ≥5cm), and the "timestamp range" and "spatial coordinate range" corresponding to this ID are recorded. Key image and trajectory matching: The timestamps of key images are extracted and matched with the timestamp sequence of the SLAM trajectory using error tolerance (allowing a time error of ±50ms; if exceeded, the timestamp is corrected through linear interpolation); after successful matching, the image is associated with the corresponding "trajectory index ID," and simultaneously, the relative coordinate offset of the target object (e.g., device, customer) in the image relative to the robot's current position is calculated using visual algorithms (e.g., feature point matching), supplementing the spatial association dimension. Matching status data with trajectories: Status data (such as temperature and vibration of industrial equipment, and interaction duration in commercial applications) is matched with the "trajectory index ID" of the SLAM trajectory by timestamp; at the same time, combined with semantic maps (pre-stored spatial locations of devices / service objects) or visual recognition (such as device nameplates and store logo positioning), the absolute coordinates of the target object corresponding to the status data in the SLAM map are determined, realizing the binding of "status data-spatial location-trajectory index".

[0065] Finally, in the storage and query phase, a structured association with a unified index is established. The three types of data are ultimately linked together using "trajectory index ID" as the core key to construct a structured association table, supporting multi-dimensional retrieval. In this embodiment, queries can be performed by "time range," where inputting a time period retrieves all trajectory index IDs within that period, and then retrieves associated key images and status data. Queries can also be performed by "spatial region," where selecting a region on the SLAM map and matching the trajectory index ID with coordinate ranges retrieves images and status data within that region. Finally, queries can be performed by "status threshold," where inputting status parameters (such as "temperature > 50℃") filters out status data that meets the conditions, and then back-linking it to the corresponding trajectory index ID and key images.

[0066] Please see Figure 5 This application also provides an association compensation mechanism for abnormal scenarios. If a certain type of data is temporarily missing (such as a short-term SLAM failure), the association continuity is maintained in the following ways: When SLAM fails, the relative pose is output by "visual odometry + semantic map", and coordinates are calculated by combining the historical SLAM map to ensure that the image and state data can still be associated with an approximate spatial location. When image acquisition fails, the state data and SLAM trajectory are still associated by timestamp. After the image is recovered, the image backtracking corresponding to "historical trajectory + state data" is supplemented (such as reconstructing the path through the SLAM trajectory to trigger image re-acquisition). When state data is missing, the SLAM trajectory and key images are still associated, and the missing state data is marked as "temporarily missing". It is recorded and associated again after the sensor is restored.

[0067] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application realize stable three-dimensional association of "SLAM trajectory-key image-state data" through the above-mentioned full-process mechanism of "acquisition synchronization-preprocessing matching-structured storage-anomaly compensation", providing a data foundation for high-precision backtracking and scene analysis.

[0068] The following is a detailed description and explanation of the solutions in the embodiments of the present invention, using specific application examples: The embodiments of this application can be applied to the field of data processing technology for humanoid robots. Taking an indoor industrial application as an example, a humanoid robot is used to conduct inspections in a precision machining workshop. Specifically, the central control system sends the command "Inspect Machines 1-10". After parsing the command, the robot command receiving module activates the multi-source sensors. The laser SLAM positioning display shows that the robot moves from the standby point (X1=5m, Y1=5m) to the No. 1 pick-and-place machine (X2=5.3m, Y2=5m), with a moving distance of 0.3m (=L0=0.3m). Data acquisition is initiated: Positioning data: Laser SLAM constructs a workshop map, the binocular camera calculates visual odometry to correct drift, and the semantic map matches the nameplate of the No. 1 machine (coordinates X=5.3, Y=5), outputting the positioning result (X=5.302, Y=5.001, accuracy ±1.5cm); Environmental video data: The binocular camera captures images of the workpiece, fixture, and nameplate; Status data: The temperature of the No. 1 machine is T=45℃ (standard value ≤50℃), vibration is f=20Hz (standard value ≤25Hz), and ΔS=0%.

[0069] Then, in this embodiment of the application, the working status of the humanoid robot is determined by inputting the following parameters: P=10%, F=2N, D o =2.0m (distance from adjacent equipment), ΔS=0%, matching fuzzy rules calculate P1=40% (<P0=95%), continue data collection; when inspecting machine tool No. 10, P=100%, F=1.8N, D o =1.8m, ΔS=2%, P1=95%, stop data acquisition. Then perform data preprocessing. By matching the positioning data with the moving speed v=0.18m / s (<v0=0.2m / s), determine T=8min after acquisition, delete the video segment after T, and obtain the target video (8min); since 8min>T0=5min, delete the first 3min and keep the last 5min. Calculate F: v=0.18m / s, ρ=1.2, d=0.5m, ε=0.8 (4 obstacles within 10m² of the workshop). 5×60×0.54≈162 sample images were extracted. Key image selection used CNN to identify the feature "workpiece position deviation of 0.5mm" in the image of machine tool No. 6 (acquired at 14:25), with C=88% (≥C0=80%), and selected as the key image; associated status data: T=48℃, f=23Hz. Backtracking revealed that: 3 days later, the fixture of machine tool No. 6 malfunctioned. Maintenance personnel initiated a backtracking process via the terminal (time 14:20-14:30, area: machine tool No. 6); the terminal displayed the robot trajectory on the SLAM workshop map (accurate to X=5.802, Y=6.003). Clicking the 14:25 trajectory node displayed the key image (workpiece deviation) and status data in a pop-up window, allowing the cause of the malfunction to be located within 15 minutes, improving efficiency by 66.7%.

[0070] This application embodiment can also be applied to indoor commercial scenarios, using a humanoid robot to guide customers in shopping malls. When a customer's voice command "Guide to Brand B shoe store (X=20m, Y=15m)" is triggered, the robot analyzes the command and starts data collection. Visual SLAM positioning shows a movement distance of 0.6m (>L0=0.5m), and the robot starts collecting positioning data: visual SLAM constructs a floor map, binocular cameras correct drift, semantic map matches the "elevator entrance LOGO" to calibrate coordinates, and the output positioning accuracy is ±2.5cm; video data: capturing the customer and store signage; status data: t=60s (expected 40s, ΔS=50%), ΔL=0.2m.

[0071] Then, the robot's operating status is determined by inputting parameters P=30%, F=0N, and D. o =1.2m, ΔS=50%, P1=30%, continue data collection; when reaching the store, P=100%, ΔS=8%, P1=88%, stop data collection.

[0072] The backtracking application identified a customer complaint about "taking a detour." Management personnel backtracked the SLAM trajectory (0.3m deviation from the planned path at the intersection of level 2, X=12.005, Y=8.002). The key image showed the customer hesitating at this point. The status data was t=120s. It was confirmed that the detour was due to the failure to identify customer behavior. Subsequent algorithm optimization improved the service accuracy to 90%.

[0073] This application embodiment can also be applied to large outdoor venues, using humanoid robots to guide materials in open-air factories. The robot moves from indoor workshop 1 (SLAM positioning, ±2cm) to an outdoor passage (GPS signal strength -90dBm), automatically switching to "SLAM + Beidou / GPS" fusion positioning. Through extended Kalman filtering, the output accuracy is ±0.3m, with a switching delay of 0.4 seconds and no lag. Then, by collecting GPS / SLAM fusion trajectory, material location video, and handling time data, during backtracking, the terminal displays a seamless connection map of "indoor SLAM trajectory + outdoor GPS trajectory," with key images marking material handover points, achieving full-process traceability.

[0074] Please see Figure 6 This application also provides a scene retracing system for a humanoid robot, which can implement the above-mentioned method. The system includes: The data acquisition module 601 is used to trigger the humanoid robot to perform multi-source data acquisition and processing according to the operation instructions, and to control the data acquisition process through the operation status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data; The sample extraction module 602 is used to perform data preprocessing and sample extraction processing on the environmental video data based on the positioning data to obtain sample images; Image filtering module 603 is used to perform target feature recognition and image filtering processing on the sample image to obtain a scene image; The visualization backtracking module 604 is used to perform visualization backtracking processing on the positioning data, the scene image, and the target object state data to obtain the scene backtracking result.

[0075] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0076] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0077] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 using the methods described in the embodiments of this application. The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0079] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0080] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0081] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0082] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0084] This application provides a method, system, device, and product for scene backtracking of a humanoid robot. The solution triggers the humanoid robot to perform multi-source data acquisition and processing based on work instructions, and controls the data acquisition process through the humanoid robot's operational status to obtain a multi-source dataset. The multi-source dataset includes positioning data, environmental video data, and target object status data. This application's embodiment controls the data acquisition process by detecting the humanoid robot's operational status, enabling on-demand data acquisition and improving data acquisition efficiency. Furthermore, this solution performs visual backtracking processing on the positioning data, scene images, and target object status data to obtain scene backtracking results. This not only enables basic data display but also allows for scene-based analysis, improving the working efficiency of the humanoid robot.

[0085] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0086] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0087] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0089] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0090] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0091] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0092] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A scene backtracking method for a humanoid robot, characterized in that, The method is applied to humanoid robots and includes the following steps: The humanoid robot is triggered to perform multi-source data acquisition and processing according to the work instructions, and the data acquisition process is controlled and processed by the work status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data; Based on the positioning data, the environmental video data is preprocessed and sample extraction is performed to obtain sample images; The sample images are subjected to target feature recognition and image filtering processing to obtain scene images; The location data, the scene image, and the target object state data are subjected to visualization backtracking processing to obtain the scene backtracking result.

2. The method according to claim 1, characterized in that, The process of controlling and processing the data acquisition through the working state of the humanoid robot to obtain a multi-source dataset includes: Sensor data is acquired through multi-source sensors; The task instruction is decomposed and progress is identified to obtain the execution progress of the task instruction; Based on the sensor data and the progress of the execution of the work instructions, the humanoid robot is subjected to probability prediction processing of its work state to obtain the work state; According to the operation status, the humanoid robot is controlled to perform anomaly detection and data acquisition and processing. The positioning data is obtained by data fusion through the synchronous positioning and map building module, the visual navigation module and the semantic map, and the environmental video data and the target object status data are collected simultaneously.

3. The method according to claim 2, characterized in that, The location data obtained by fusing data through a synchronous positioning and map building module, a visual navigation module, and a semantic map includes: The environmental map is obtained by performing environmental map construction and dynamic obstacle filtering through the synchronous positioning and map building module. The visual navigation module collects environmental feature points and performs trajectory correction processing on the environmental map based on the environmental feature points to obtain trajectory data. The location data is obtained by performing positioning calibration processing on the trajectory data using the landmark data of the semantic map.

4. The method according to claim 1, characterized in that, The step of performing data preprocessing and sample extraction processing on the environmental video data based on the positioning data to obtain sample images includes: Based on the positioning data, the humanoid robot's movement speed is matched and time is recorded to obtain the data clipping start timestamp. Based on the data cropping start timestamp, redundant segments are removed from the environmental video data to obtain the target video data. The target video data is processed by image frame extraction according to a preset sampling frequency to obtain the sample image.

5. The method according to claim 1, characterized in that, The process of performing target feature recognition and image filtering on the sample image to obtain a scene image includes: The sample image is subjected to target feature recognition processing to obtain target features; The target features are input into a convolutional neural network for confidence assessment to obtain the target confidence score. The sample images are filtered based on the target confidence level to obtain the scene image.

6. The method according to any one of claims 1 to 5, characterized in that, The visualization backtracking process of the positioning data, the scene image, and the target object state data to obtain the scene backtracking result includes: The positioning data, the scene image, and the target object state data are correlated and mapped to obtain mapped data. The mapped data is subjected to fault feature extraction and fusion processing to obtain fault features; The fault characteristics are subjected to fault prediction and root cause analysis to obtain the scenario backtracking results.

7. The method according to claim 6, characterized in that, The process of associating and mapping the positioning data, the scene image, and the target object state data to obtain mapping data includes: The location data, the scene image, and the target object status data are time-stamped and spatially bound to obtain bound data. The bound data is subjected to trajectory index generation and matching processing to obtain matching data; A structured association table is constructed based on the matching data, and anomaly compensation processing is performed on the structured association table to obtain the mapping data.

8. A scene retracing system for a humanoid robot, characterized in that, The system includes: The data acquisition module is used to trigger the humanoid robot to perform multi-source data acquisition and processing according to the operation instructions, and to control the data acquisition process through the operation status of the humanoid robot to obtain a multi-source dataset; the multi-source dataset includes positioning data, environmental video data and target object status data; The sample extraction module is used to perform data preprocessing and sample extraction processing on the environmental video data based on the positioning data to obtain sample images; The image filtering module is used to perform target feature recognition and image filtering processing on the sample images to obtain scene images; The visualization backtracking module is used to perform visualization backtracking processing on the positioning data, the scene image, and the target object status data to obtain the scene backtracking result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images