Production robot control system based on semantic recognition

Abstract

The application discloses a production robot control system based on semantic recognition, and relates to the technical field of robot control.The system comprises an environment information acquisition module, which is used for acquiring environment information;an environment semantic recognition and trajectory analysis module, which is used for identifying the category of static objects and the moving trajectory of dynamic objects in the environment information; a semantic map construction module, which is used for constructing a 3D semantic map; an operation information acquisition module, which is used for acquiring operation information input by an operator; a semantic information extraction and analysis module, which is used for extracting action semantics and voice semantics; a credibility evaluation and fusion module, which is used for generating final semantic information; a control instruction generation module, which is used for constructing a control instruction; a path planning module, which is used for generating a task path; and an action execution module, which is used for controlling the robot to execute a task according to the task path.The application has the advantages of dynamic environment adaptation, strong production scene pertinence and reliable multi-source semantic fusion.

Description

Semantic Recognition-Based Production Robot Control System Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a production robot control system based on semantic recognition. Background Technology

[0002] Robots have been widely used in many fields, especially in factories. They can independently complete tasks that previously required multiple people to work together, and their efficiency and safety are far superior to those performed by humans. Therefore, robots have replaced human workers in some dangerous and repetitive jobs.

[0003] Although robots have standardized procedures established before execution based on task requirements and environmental conditions, the actual environment can change rapidly. This necessitates on-site control by operators to prevent the robot from operating according to the original procedures and causing potential hazards. Currently, common control methods include voice and motion. Robots use microphones and cameras to collect voice and motion data from personnel, understand their requests, and execute corresponding actions. However, this control method is primarily used for service robots, such as guide robots in public places or companion robots in homes. These robots face relatively simple scenarios and tasks, significantly different from production scenarios. Production scenarios involve numerous personnel and equipment in various positions and with varying roles, and these personnel and equipment may move at any time. This requires robots to not only understand the intentions conveyed by operators through voice or motion but also consider real-time environmental changes to ensure accurate and effective robot control. Current robot control technology cannot yet meet these requirements. Summary of the Invention

[0004] This application provides a production robot control system based on semantic recognition to solve the problem that existing technologies cannot take into account both operator intentions and environmental changes in complex and ever-changing production environments.

[0005] This application provides a semantic recognition-based production robot control system, including:

[0006] The environmental information acquisition module is used to collect environmental information about the environment in which the robot is located.

[0007] The environmental semantic recognition and trajectory analysis module is used to identify the categories of static objects in environmental information using deep learning semantic segmentation and object detection algorithms, and obtain the category label of each static object; it uses optical flow combined with inter-frame difference to identify dynamic objects in environmental information and obtain the movement trajectory of dynamic objects; it summarizes the patterns of the movement trajectory of dynamic objects to form a semantic attribute library of dynamic objects.

[0008] The semantic map construction module is used to construct a sparse point cloud map based on environmental information using the ORB-SLAM3 algorithm. It combines the category labels of static objects with the sparse point cloud map to form a static 3D semantic map; and it overlays the movement patterns of dynamic objects in the semantic attribute library of dynamic objects onto the static 3D semantic map to form a 3D semantic map.

[0009] The operation information acquisition module is used to collect the operation information input by the operator, including action information and voice information;

[0010] The semantic information extraction and parsing module is used to extract the semantic meaning of the operator's hand movements from the motion information using a hand pose estimation algorithm, and to extract the semantic meaning of the speech information using natural language processing technology.

[0011] The credibility assessment and fusion module is used to score the credibility of action semantics and speech semantics from signal quality indicators, environment matching indicators and behavior consistency indicators, and obtain the corresponding credibility scores. Based on the values ​​of the credibility scores of action semantics and speech semantics, it determines whether to select action semantics or speech semantics, or both, to obtain the final semantic information.

[0012] The control instruction generation module is used to construct control instructions based on the final semantic information;

[0013] The path planning module is used to generate a task path based on the target position in the control command and the robot's current position using the A* algorithm. The task path avoids static objects in the 3D semantic map while also maintaining a safe distance from dynamic objects.

[0014] The motion execution module is used to control the robot to perform tasks according to the task path.

[0015] The semantic recognition-based production robot control system in this application has the following advantages:

[0016] 1. Dynamic environment adaptability. Dynamic 3D semantic maps not only update the position of objects in real time, but also summarize movement patterns and predict future states, effectively coping with the frequent movement of personnel, materials, and tools in the production environment and significantly reducing the risk of collisions.

[0017] 2. Reliability of Multi-Source Semantic Fusion. A credibility assessment mechanism filters action and speech semantics, addressing recognition errors caused by interference from a single modality and ensuring the accuracy and stability of control commands.

[0018] 3. Highly targeted to production scenarios. With 3D semantic maps (collision avoidance) and production tasks (action specifications) as dual constraints, it is adapted to the high precision and high safety requirements of production robots and can be widely used in various production scenarios such as assembly, handling, and inspection.

[0019] 4. Closed-loop control ensures accuracy. From instruction generation, path planning, action execution to result verification, a complete closed loop is formed. Deviations are corrected through a real-time adjustment mechanism to ensure the accuracy of action execution and the quality of task completion. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 is a block diagram of the semantic recognition-based production robot control system provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Figure 1 is a block diagram of the semantic recognition-based production robot control system provided in an embodiment of this application. This embodiment of the application provides a semantic recognition-based production robot control system, including:

[0024] The environmental information acquisition module is used to collect environmental information about the environment in which the robot is located.

[0025] The environmental semantic recognition and trajectory analysis module is used to identify the categories of static objects in environmental information using deep learning semantic segmentation and object detection algorithms, and obtain the category label of each static object; it uses optical flow combined with inter-frame difference to identify dynamic objects in environmental information and obtain the movement trajectory of dynamic objects; it summarizes the patterns of the movement trajectory of dynamic objects to form a semantic attribute library of dynamic objects.

[0026] The semantic map construction module is used to construct a sparse point cloud map based on environmental information using the ORB-SLAM3 algorithm. It combines the category labels of static objects with the sparse point cloud map to form a static 3D semantic map; and it overlays the movement patterns of dynamic objects in the semantic attribute library of dynamic objects onto the static 3D semantic map to form a 3D semantic map.

[0027] The operation information acquisition module is used to collect the operation information input by the operator, including action information and voice information;

[0028] The semantic information extraction and parsing module is used to extract the semantic meaning of the operator's hand movements from the motion information using a hand pose estimation algorithm, and to extract the semantic meaning of the speech information using natural language processing technology.

[0029] The credibility assessment and fusion module is used to score the credibility of action semantics and speech semantics from signal quality indicators, environment matching indicators and behavior consistency indicators, and obtain the corresponding credibility scores. Based on the values ​​of the credibility scores of action semantics and speech semantics, it determines whether to select action semantics or speech semantics, or both, to obtain the final semantic information.

[0030] The control instruction generation module is used to construct control instructions based on the final semantic information;

[0031] The path planning module is used to generate a task path based on the target position in the control command and the robot's current position using the A* algorithm. The task path avoids static objects in the 3D semantic map while also maintaining a safe distance from dynamic objects.

[0032] The motion execution module is used to control the robot to perform tasks according to the task path.

[0033] For example, the environmental information acquisition module includes an RGB camera, a LiDAR, a depth camera, and an inertial measurement unit. The environmental information includes environmental color images, LiDAR point cloud data, depth images, and pose information.

[0034] Specifically, RGB cameras can acquire visual features such as the appearance, color, and texture of objects by collecting environmental color images; LiDAR can acquire the three-dimensional contours, distance information, and spatial distribution of objects by collecting laser point cloud data; depth cameras can supplement the details of the relative distance between objects and robots by collecting depth images; and inertial measurement units (IMUs) can provide a reference for map positioning by collecting the robot's own pose information (three-axis attitude angles and acceleration).

[0035] Furthermore, it includes a preprocessing module for preprocessing environmental information. For environmental color images, the preprocessing module removes noise caused by dust and light variations in the production environment using noise reduction algorithms (such as Gaussian filtering). For laser point cloud data, the preprocessing module simplifies the data volume and removes isolated points to eliminate interference through downsampling (voxel mesh method). For depth images, the preprocessing module performs pixel-level registration between the depth image and the environmental color image to align spatial coordinates. For pose information, the preprocessing module corrects drift using a Kalman filter algorithm to ensure the accuracy of pose data. The preprocessed data is uniformly converted into a standardized data format in the robot's base coordinate system, providing a foundation for subsequent modeling.

[0036] For static objects, deep learning semantic segmentation and object detection algorithms, such as YOLOv5, are used to process the data after the registration of the environmental color image and the fusion of the laser point cloud to identify the categories of static objects in the production environment, including production equipment (such as machine tools and conveyor belts), tooling fixtures (such as clamps and pallets), materials (such as raw materials, semi-finished products, and finished products), and fixed facilities (such as columns and walls). The system outputs the category label (such as "Machine Tool A", "Aluminum Profile Pallet", "Semi-finished Gear"), two-dimensional bounding box, and three-dimensional coordinate range for each static object.

[0037] For dynamic objects, optical flow combined with inter-frame difference analysis is used to identify dynamic objects, including production personnel, mobile tool vehicles, and transfer robots, from continuously acquired image sequences and point cloud data. For each dynamic object, its three-dimensional coordinates are extracted from continuous timestamps, such as 10 frames of data within 1 second, to form raw trajectory data. The raw trajectory data is smoothed using a sliding window method to remove outliers caused by sudden interference, resulting in a stable movement trajectory of the dynamic object.

[0038] Historical trajectory data of dynamic objects, such as trajectory data from the past hour, is analyzed over time to summarize movement patterns. Specifically, for periodically moving objects, such as regularly patrolling personnel and cyclically transporting vehicles, features such as movement cycle, repeated paths, and dwell time are extracted. For non-periodic moving objects, such as temporary workers and emergency material transport, features such as movement speed range, activity area distribution, and common movement directions are statistically analyzed. These patterns and features are then bound to object category and current location to form a semantic attribute library for dynamic objects.

[0039] The semantic map construction module, based on preprocessed laser point clouds and depth images, uses the ORB-SLAM3 algorithm to construct a sparse point cloud map of the environment. Combined with robot pose information provided by the IMU, it achieves global localization and stitching of the map. The module integrates the category labels and 3D coordinate ranges of static objects obtained from semantic recognition with the sparse point cloud map, assigning corresponding semantic attributes to each point cloud cluster to form the basic framework of a static 3D semantic map.

[0040] Then, the semantic map construction module overlays the current position, category label, and movement patterns of dynamic objects onto the static 3D semantic map in real time. Different colors are used to distinguish between dynamic and static objects in the map, dashed boxes are used to mark the predicted activity areas of dynamic objects, and arrows are used to indicate common movement directions, forming an initial 3D semantic map. The map contains three core levels: a geometric level (including the object's 3D coordinates and outline), a semantic level (including the object's category and attributes), and a dynamic level (including the movement patterns and predicted areas of dynamic objects).

[0041] Furthermore, it also includes a semantic map update module, which is used to initiate the update of the 3D semantic map according to a set update cycle or when the lidar or depth camera detects a static object or a moving object.

[0042] Specifically, if a static object is detected to have moved, or if a static object is added or removed, the semantic map update module re-identifies the semantic attributes and 3D coordinates of the static object and replaces the semantic and geometric information of the corresponding area in the 3D semantic map with the re-identified semantic attributes and 3D coordinates. For dynamic objects, the semantic map update module updates the current position of the dynamic object in real time and corrects the movement pattern based on the latest movement trajectory.

[0043] Furthermore, two types of update trigger conditions are set, and map updates are initiated when either condition is met. The first is timed updates, with an update cycle of 0.5 seconds to adapt to the rapid movement of objects in the production environment; the second is event-triggered updates, which are initiated immediately when the LiDAR or depth camera detects an object position change exceeding a threshold (e.g., 5cm), when a new / disappearing object is added, or when the movement pattern of a dynamic object deviates from historical characteristics, such as a deviation exceeding 30%.

[0044] For static objects, if movement (e.g., tooling fixture adjustment), addition (e.g., material replenishment), or removal (e.g., finished product shipment) is detected, their semantic attributes and 3D coordinates are re-identified, and the corresponding semantic and geometric information in the map is replaced. For dynamic objects, their current position is updated in real time, and movement patterns are corrected based on the latest trajectory data, adjusting the predicted activity area and direction arrows. Finally, combining the robot pose data from the IMU and the global positioning information from the LiDAR, the updated map is calibrated to ensure map stitching accuracy in different areas and avoid cumulative errors.

[0045] Meanwhile, the semantic map update module also automatically stores dynamic update records from the past 24 hours, including object position change logs, dynamic object movement pattern iteration processes, map calibration parameters, etc., to trace the trajectory of environmental changes and provide historical data support for subsequent semantic understanding and collision risk assessment.

[0046] In the embodiments of this application, the robot is equipped with a high-definition gesture camera (30fps), mounted on the robot's shoulder and hand end, ensuring coverage of frequently used areas for human operation and an area 0.5-3 meters away from the robot. The camera captures the human's body movements, focusing on hand movements (gestures, grasping / releasing postures) and torso postures (such as pointing gestures, start / stop gestures), and simultaneously records the timestamps of the movements and the relative distance between the human and the robot. During the acquisition process, image cropping technology can be used to focus on the area of ​​human movement to eliminate background interference from production equipment, materials, etc.

[0047] Meanwhile, three omnidirectional microphones are installed near the robot's head and operating area, forming a microphone array to collect voice commands from the operator. The microphone array supports noise reduction algorithms (such as adaptive beamforming) to suppress mechanical noise and equipment operating noise in the production environment (such as machine tool noise and conveyor belt motor noise), improving the clarity of the voice signal. The collected voice signal is converted into digital audio data with a sampling rate of 16kHz to ensure the accuracy of semantic extraction.

[0048] For the collected action information, the semantic information extraction and parsing module uses a hand pose estimation algorithm based on the MediaPipe framework to extract the three-dimensional coordinates of 21 joints of the operator's hand. Combined with limb contour analysis, it achieves two types of action semantic parsing. First, gesture semantic recognition: for predefined commonly used gestures in production scenarios, such as a clenched fist for "start," a shrug for "pause," an index finger pointing for "move to target," a thumb pointing upwards for "accelerate," a thumb pointing downwards for "decelerate," and a two-handed grasping action for "clamping materials," a deep learning classification model (based on ResNet50) is used to identify gesture categories and output gesture semantic labels, such as "start," "move to target," and "clamping materials." Second, action intent parsing: continuous action sequences are analyzed. For example, if a person points to a piece of equipment and then makes a grasping gesture, it is parsed as "grabbing materials next to the equipment"; if a person makes a pause gesture and then points to a fault area, it is parsed as "pausing the current action and heading to the fault area."

[0049] For voice information, the semantic information extraction and parsing module uses Automatic Speech Recognition (ASR) technology to convert audio data into text, removing redundant information such as interjections and repetitive sentences to obtain clean text instructions. Semantic parsing is then performed using Natural Language Processing (NLP) algorithms to extract core instruction elements: action type (e.g., "move," "assemble," "transfer," "pause"), target object (e.g., "material number 3," "clamp of machine tool 2"), action parameters (e.g., "speed 5cm / s," "accuracy ±0.1mm," "deliver to assembly station B"), and time requirements (e.g., "immediately," "within 3 minutes"). For example, the voice instruction "deliver the semi-finished gear on the left pallet to the assembly station of machine tool 3, at a slower speed" yields the following semantic elements after parsing: action type "transfer," target object "left pallet - semi-finished gear," target location "machine tool 3 - assembly station," and action parameter "low speed."

[0050] Furthermore, it also includes a standardization module, which is used to convert action semantics and speech semantics into a unified standardized semantic format, including action type, target object, target location, action parameters, and triggering conditions. For missing fields, they are supplemented according to the default values ​​of the production scenario. For example, if the speed is not specified, the default speed of the production task is used; if the target location is not specified, the default location is the nearest reasonable location near the current robot.

[0051] Furthermore, this application embodiment uses three-dimensional evaluation indicators to score the credibility of action semantics and speech semantics respectively, with a maximum score of 100 points.

[0052] Signal quality indicators (weight 40%): The signal quality of action semantics is scored based on the clarity of the gesture image, the completeness of joint recognition, and the continuity of the action. Full marks are awarded for images without blurring, joint recognition rate ≥90%, and continuous and uninterrupted action; otherwise, points are deducted proportionally. The signal quality of speech semantics is scored based on the signal-to-noise ratio, speech clarity, and text conversion accuracy. Full marks are awarded for a signal-to-noise ratio ≥30dB, clarity ≥85%, and conversion accuracy ≥90%.

[0053] Environmental matching index (weight 30%): assesses the degree of matching between semantic information and the current production environment and robot status. Action semantics needs to match the relative position of the person and the target object (e.g., the target object being pointed to is within the robot's reach range) and the feasibility of the action (e.g., the weight of the material corresponding to the grasping action is within the robot's load range); speech semantics needs to match objects existing in the environment (e.g., the mentioned "machine tool 5" actually exists) and task compatibility (e.g., the "assembly" action is consistent with the current production task). The higher the matching degree, the higher the score.

[0054] Behavioral consistency index (weight 30%): Analyzes the recent behavioral patterns of personnel. If the current action / voice is consistent with the operation intention in the past minute, such as pointing to the same material twice in a row and issuing a handling instruction, the score is higher. If there is a contradiction, such as making a pause gesture and issuing a start voice at the same time, the corresponding semantic consistency score is lower.

[0055] Calculate the action semantic credibility score S1 and the speech semantic credibility score S2 respectively, and set a threshold T=60 points. Scores below the threshold are considered untrustworthy.

[0056] If S1≥60 and S2<60, action semantics are used to generate control commands, ignoring untrusted speech semantics. If S2≥60 and S1<60, speech semantics are used to generate control commands, ignoring untrusted action semantics.

[0057] If S1≥60 and S2≥60, and the semantic elements of both are completely identical (e.g., action pointing to material A + voice "transfer material A"), then a control command is generated by fusion, strengthening the execution priority. If the core elements are identical but the secondary parameters differ (e.g., action pointing to material A + voice "transfer material A to assembly table B, speed 5cm / s"), and the action does not specify speed and position, the action semantics are supplemented with the voice semantic parameters to generate a fused command. If the core elements conflict (e.g., action pointing to material A + voice "transfer material B"), the semantic with the higher score can be used as the primary command, while the other semantic is used as an auxiliary reference. If the auxiliary reference conflicts with the primary command, this can be resolved through environmental verification. For example, if material A is confirmed to be required for the current task through a 3D semantic map, then the primary command is corrected; if verification is not possible, command generation is paused, and the operator is consulted via robot voice for confirmation.

[0058] If S1 < 60 and S2 < 60, pause instruction generation and the robot will issue a prompt, such as "No valid control instruction was recognized. Please operate or give a voice command again." At the same time, it will continue to collect action and voice information until reliable semantics are obtained.

[0059] Furthermore, the control command generation module constructs structured control commands based on the final semantic information, namely action semantics, voice semantics, or fused semantics, according to the control protocol of the production robot. The control commands include a head identifier (command type), core parameters (action type, target object ID, target position coordinates, action speed, accuracy requirement, execution order), constraint identifiers (collision avoidance, task compatibility), and a tail checksum (ensuring command transmission integrity). For example, the control command corresponding to fused semantics includes: a head identifier "transfer command", core parameters "action type = transfer, target object ID = material A-003, target position coordinates = (X:1500mm, Y:800mm, Z:500mm), speed = 5cm / s, accuracy = ±0.2mm, execution order = 1", constraint identifiers "collision avoidance = enabled, task compatibility = assembly task", and tail checksum = XXX.

[0060] Furthermore, after constructing the control instructions, the control instruction generation module verifies their validity. The verification includes whether the target exists and is reachable in the 3D semantic map, whether the motion parameters are within the robot's hardware limits (such as load, speed, and travel distance), and whether the execution order conforms to the production logic. If the verification passes, the final control instructions are output. If the verification fails, the control instruction generation module feeds back to the semantic fusion stage to adjust the semantic parameters, such as reducing excessive speed parameters and correcting unreachable target positions, in order to regenerate the control instructions.

[0061] Furthermore, it also includes a feasibility analysis module, which is used for collision risk assessment based on 3D semantic maps and action constraint verification based on production tasks to determine whether control instructions are feasible. If the control instructions are not feasible, the feasibility analysis module adjusts the action path, execution timing or action parameters in the control instructions, and then performs the feasibility analysis again until the control instructions are feasible.

[0062] Specifically, collision risk assessment based on 3D semantic maps involves overlaying and analyzing the target position and motion path in control commands with a dynamic 3D semantic map. A bounding box algorithm can be used to construct a volumetric model of the robot's motion, including the robot body, end effector, and gripped materials. Collision detection is then performed between this model and static objects, dynamic objects, and their predicted activity areas on the map. If the distance between the motion path and an object is less than a safety threshold (5cm by default in production scenarios and 2cm by default in precision operation scenarios), a collision risk is identified. For dynamic objects, their movement patterns are used to predict their position changes within the next second. If the robot's motion intersects with the predicted position of the dynamic object, a collision risk is also identified.

[0063] Motion constraint verification based on production tasks involves retrieving a pre-stored production task parameter library. This library contains the motion specifications for the current task, such as the motion sequence for assembly tasks ("grab-move-position-assemble-release"), accuracy requirements (±0.1mm), speed limits (≤3cm / s), and material clamping force (5-8N); and the path priority (preferential use of preset channels), obstacle avoidance priority (personnel > equipment > materials), and placement posture (horizontal placement, tilt angle ≤3°) for handling tasks. The verification checks whether the motion parameters in the control commands meet these constraints, such as whether the assembly motion accuracy meets the standards, whether the handling path deviates from the preset channel, and whether the clamping force is within a reasonable range.

[0064] If the collision risk assessment is negative and the action constraint verification passes, the action is deemed feasible, and the path planning phase begins. If a collision risk exists or the action constraint is violated, the action is deemed infeasible, and an adjustment mechanism is activated: for collision risks, the action path or execution timing is adjusted, such as waiting for the dynamic object to leave the prediction area; for violations of action constraints, the action parameters are corrected, such as reducing speed or adjusting clamping force, and a feasibility analysis is re-performed until the requirements are met.

[0065] In the embodiments of this application, the path planning module generates an initial path using the A* algorithm based on the target position of the control command and the robot's current position. The objective function of path planning is "shortest path + least collision risk + highest task compatibility," with weights set to 0.3, 0.5, and 0.2, respectively. The initial path must avoid static obstacles in the 3D semantic map while maintaining a safe distance from the current position of dynamic objects.

[0066] Furthermore, after generating the task path, the path planning module optimizes the path by combining real-time updates from the 3D semantic map. During the optimization process, if a change in the position of a static or dynamic object is detected, a local path replanning algorithm (such as the D* Lite algorithm) is used to adjust the path segments to ensure no collision risk throughout the process. Simultaneously, path parameters are adjusted according to production task constraints. For example, the path for a precision assembly task needs to be smooth without sharp turns to reduce robot vibration; the path for a high-speed transport task prioritizes straight lines to shorten execution time.

[0067] After completing the task path planning, the path planning module can also mark key nodes on the optimized path, including the starting point (the robot's current position), obstacle avoidance points (turning points to bypass obstacles), positioning points (precise positioning positions close to the target object), and the ending point (the target position). Each key point is marked with corresponding action parameters, such as the speed limit for obstacle avoidance points and the accuracy requirements for positioning points, providing precise guidance for action execution.

[0068] Furthermore, during the execution of tasks according to the task path, the action execution module receives updated data from the 3D semantic map. If new obstacles or dynamic objects enter the task path, the current task is paused and the local path is replanned. If the target object is removed or the production equipment malfunctions, the task execution is paused and a prompt message is sent to the operator.

[0069] Specifically, the robot control system drives the moving mechanism (such as a caster wheel) and the actuator (such as a robotic arm) to perform actions based on the key points and motion parameters of the task path. The moving mechanism travels along the planned path, using IMU and LiDAR for real-time positioning to ensure driving accuracy. The actuator operates according to preset action types (such as grasping, assembling, and placing), using force sensors to provide feedback on clamping force and vision sensors to provide feedback on positioning deviations, thus achieving closed-loop control. For example, during a grasping action, when the force sensor detects that the clamping force has reached a preset value (such as 6N), the clamping stops; when the vision sensor detects a material positioning deviation (such as a deviation of 0.15mm in the X direction), the position of the robotic arm is finely adjusted to correct the deviation.

[0070] During action execution, the action execution module also simultaneously receives updated data from the 3D semantic map. If the map shows a slight change in the position of the target object, such as a deviation ≤0.5cm, the position of the robot's end effector is corrected in real time through the vision guidance system. If the map shows a new obstacle or a dynamic object entering the movement path, the emergency obstacle avoidance program is immediately activated, the current action is paused, and the local path is replanned before continuing execution. If the map shows a significant change in the environment, such as the target object being removed or production equipment malfunctioning, the action execution is paused, the robot issues a prompt message, and waits for personnel confirmation before proceeding with subsequent operations.

[0071] During the execution of actions, the action execution module continuously verifies the conformity of action parameters with production task constraints. If environmental interference causes the action accuracy to deviate from the preset value, such as an assembly accuracy deviation of 0.2mm exceeding the constraint threshold of 0.1mm, the action speed will be automatically reduced and the number of positioning attempts will be increased until the accuracy meets the standard. If the action execution sequence violates the task logic, such as starting movement before the grab is completed, the action execution module will automatically pause and backtrack to the previous critical node to re-execute the correct sequence.

[0072] After the action is completed, the execution effect can be detected by multimodal sensors. The vision sensor captures images of the target area and identifies whether the position and posture of the target object meet the task requirements, such as whether the material is accurately placed in the target position and whether the assembly is in place; the force sensor detects the force state of the actuator, such as whether the fitting force after assembly is within a reasonable range; the 3D semantic map updates the latest state of the target object and confirms that its position and posture are consistent with the task requirements.

[0073] Based on the detection results, task completion can be evaluated from three dimensions: positional accuracy (the deviation between the actual position of the target object and the preset position), posture accuracy (the deviation between the actual posture of the target object and the preset posture), and task integrity (whether all instruction requirements have been fulfilled, such as whether the material has been delivered to the designated location and released in a handling task). If all three dimensions meet preset thresholds, such as positional accuracy ≤ 0.2mm, posture accuracy ≤ 2°, and task integrity = 100%, the task is considered complete; if any dimension is not met, the task is considered incomplete.

[0074] If the task is completed, the robot updates its status to "ready" and feeds back the execution result to the production management system. At the same time, it updates the status of the target object in the 3D semantic map, such as "moved to assembly station B". If the task is not completed, the robot analyzes the reason for the failure, such as excessive positioning deviation or environmental interference causing the action to be interrupted. It automatically starts the retry mechanism and optimizes the action parameters during the retry process, such as increasing the positioning time and adjusting the obstacle avoidance strategy. If the task is still not completed after 3 retries, the robot issues a fault prompt and waits for personnel to investigate and handle it. At the same time, it records the fault information, including time, location and cause, for subsequent analysis.

[0075] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0076] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A production robot control system based on semantic recognition, characterized in that, include: The environmental information acquisition module is used to collect environmental information about the environment in which the robot is located. The environmental semantic recognition and trajectory analysis module is used to identify the category of static objects in the environmental information using deep learning semantic segmentation and object detection algorithms, and obtain the category label of each static object; it uses optical flow combined with inter-frame difference to identify dynamic objects in the environmental information and obtain the movement trajectory of the dynamic objects; it summarizes the patterns of the movement trajectory of the dynamic objects to form a dynamic object semantic attribute library. A semantic map construction module is used to construct a sparse point cloud map based on the environmental information using the ORB-SLAM3 algorithm, and combine the category labels of static objects with the sparse point cloud map to form a static 3D semantic map; the movement patterns of dynamic objects in the dynamic object semantic attribute library are superimposed on the static 3D semantic map to form a 3D semantic map; an operation information acquisition module is used to collect operation information input by the operator, including action information and voice information; a semantic information extraction and parsing module is used to extract the action semantics of the operator's hand in the action information using a hand pose estimation algorithm, and to extract the voice semantics of the voice information using natural language processing technology; a credibility assessment and fusion module is used to score the credibility of the action semantics and the voice semantics based on signal quality indicators, environmental matching indicators, and behavioral consistency indicators, and to obtain corresponding credibility scores, determine whether to select the action semantics or the voice semantics, or both, based on the values ​​of the action semantic credibility scores and the voice semantic credibility scores, to obtain the final semantic information; a control command generation module is used to construct control commands based on the final semantic information; The path planning module is used to plan the path based on the target position in the control command and the robot's current position, using A... The algorithm generates a task path that avoids static objects in the 3D semantic map while maintaining a safe distance from dynamic objects; the action execution module is used to control the robot to perform tasks according to the task path.

2. The production robot control system based on semantic recognition according to claim 1, characterized in that, The environmental information acquisition module includes an RGB camera, a LiDAR, a depth camera, and an inertial measurement unit. The environmental information includes environmental color images, LiDAR point cloud data, depth images, and pose information.

3. The production robot control system based on semantic recognition according to claim 2, characterized in that, It also includes a preprocessing module, which is used to preprocess the environmental information; for the environmental color image, the preprocessing module removes noise from the production environment through a noise reduction algorithm; for the laser point cloud data, the preprocessing module simplifies the data volume and removes isolated points through downsampling; for the depth image, the preprocessing module performs pixel-level registration between the depth image and the environmental color image; for the pose information, the preprocessing module corrects drift through a Kalman filter algorithm.

4. The production robot control system based on semantic recognition according to claim 2, characterized in that, It also includes a semantic map update module, which is used to update the 3D semantic map according to a set update cycle or when the lidar or the depth camera detects that a static object or a dynamic object is moving.

5. The production robot control system based on semantic recognition according to claim 4, characterized in that, If the position of a static object is detected to have moved, or if a static object is added or removed, the semantic map update module re-identifies the semantic attributes and three-dimensional coordinates of the static object, and replaces the semantic and geometric information of the corresponding area in the 3D semantic map with the re-identified semantic attributes and three-dimensional coordinates. For dynamic objects, the semantic map update module updates the current position of the dynamic object in real time and corrects the movement pattern based on the latest movement trajectory.

6. The production robot control system based on semantic recognition according to claim 1, characterized in that, It also includes a standardization module, which is used to convert the action semantics and the speech semantics into a unified standardized semantic format, including action type, target object, target location, action parameters and triggering conditions.

7. The production robot control system based on semantic recognition according to claim 1, characterized in that, After constructing the control instructions, the control instruction generation module verifies the validity of the control instructions. The verification includes whether the target exists and is reachable in the 3D semantic map, whether the action parameters are within the hardware limits of the robot, and whether the execution order conforms to the production logic. If the verification fails, the control instruction generation module returns to the semantic fusion stage to adjust the semantic parameters and regenerate the control instruction.

8. The production robot control system based on semantic recognition according to claim 1, characterized in that, It also includes a feasibility analysis module, which is used to assess the collision risk based on the 3D semantic map and to determine whether the control command is feasible based on the action constraint verification of the production task. If the control command is not feasible, the feasibility analysis module adjusts the action path, execution timing or action parameters in the control command, and then performs the feasibility analysis again until the control command is feasible.

9. The production robot control system based on semantic recognition according to claim 1, characterized in that, After generating the task path, the path planning module optimizes the task path by combining the real-time update information of the 3D semantic map. During the optimization process, if the position of a static or dynamic object changes, a local path replanning algorithm is used to adjust the path segment. At the same time, the path parameters are adjusted according to the production task constraints.

10. The production robot control system based on semantic recognition according to claim 1, characterized in that, During the execution of the task according to the task path, the action execution module receives updated data from the 3D semantic map. If a new obstacle or dynamic object enters the task path, the current task is paused and the local path is replanned. If the target object is removed or the production equipment malfunctions, the task execution is paused and a prompt message is sent to the operator.

Images