AR (Augmented Reality) assisting and scene sensing system for man-machine cooperation power battery disassembly

Abstract

The invention relates to an AR auxiliary and scene sensing system for man-machine cooperation power battery disassembly. The AR auxiliary and scene sensing system comprises a scene sensing module and an AR auxiliary man-machine cooperation disassembly module. The scene sensing module comprises an instance segmentation model, a posture recognition model and an action recognition model; inputting human body action behaviors and scene information videos in the power battery disassembly scene into an instance segmentation model, a posture recognition model and an action recognition model at the same time, wherein the instance segmentation model, the posture recognition model and the action recognition model independently process the received videos; the processing results of the instance segmentation model, the posture recognition model and the action recognition model are input into the AR-assisted man-machine cooperation disassembly module, and the AR-assisted man-machine cooperation disassembly module performs analogue simulation on the man-machine cooperation disassembly process by using the AR technology according to the received information, so that disassembly information visualization and man-machine safety distance judgment are realized. According to the invention, efficient and safe man-machine cooperation power battery disassembly is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of power battery disassembly, and relates to an AR assistance and scene perception system for human-machine collaborative power battery disassembly. Background Art

[0002] The current power battery recycling still uses manual or semi-automatic disassembly methods. The existing technology relies on manually controlled robotic arms to assist in assisted disassembly. However, the robotic arms cannot autonomously perform corresponding disassembly operations based on the actual disassembly situation on site. Therefore, the existing disassembly technology faces challenges such as low disassembly efficiency, non-standard disassembly and poor safety.

[0003] Faced with this challenge, AR (augmented reality) technology and scene perception technology provide new solutions for power battery removal. Scene perception technology enables the system to make intelligent decisions, respond quickly, and enhance user personalized experience and safety by capturing and processing environmental information in real time. It is widely used in autonomous driving, smart home, virtual reality, robotics, security monitoring, smart medical care, and smart retail.

[0004] In the field of autonomous driving, scene perception technology uses sensors such as cameras, radars, and lidars to help autonomous vehicles perceive road conditions, pedestrians, other vehicles, and other information in real time, thereby achieving autonomous driving decisions and safe obstacle avoidance. In smart homes, scene perception technology uses sensors to collect home environment information such as temperature, light, and sound, thereby automatically adjusting lighting, air conditioning, security systems, and other equipment to improve the user's living comfort and safety.

[0005] Scene perception technology provides a more immersive experience in virtual reality by tracking the user's position, movements, and changes in the surrounding environment, allowing the virtual world to seamlessly connect with the real world. In the field of robotics, scene perception technology enables robots to understand and navigate complex environments, such as avoiding obstacles in a factory, or identifying and performing specific tasks in a home environment. For power battery disassembly, scene perception technology can analyze the on-site environment in real time, such as spatial layout, obstacle location, etc., to guide the robot arm or technicians to perform precise and efficient disassembly operations.

[0006] AR technology can integrate virtual objects with real scenes, allowing users to see virtual objects in real scenes and present information in a more intuitive, vivid and three-dimensional way, thereby improving the effectiveness of information communication and helping users complete work tasks more efficiently; AR technology also enhances the interactivity between users and virtual objects by providing richer and more diverse interaction methods.

[0007] AR technology can help users operate in a safe environment and avoid safety accidents by providing a virtual rehearsal environment. However, the existing technology has not yet used AR technology and scene perception technology for power battery disassembly.

[0008] Therefore, it is of great significance to combine AR technology with scene perception technology to study an AR-assisted and scene perception system for human-machine collaborative power battery disassembly to solve the problems of low efficiency and poor safety in the power battery recycling process of the existing technology. Summary of the invention

[0009] The purpose of the present invention is to solve the problems existing in the prior art and provide an AR assistance and scene perception system for human-machine collaborative power battery disassembly.

[0010] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] An AR-assisted and scene-perception system for human-machine collaborative power battery disassembly, comprising a scene-perception module and an AR-assisted human-machine collaborative disassembly module;

[0012] In the scene semantic recognition module, a method combining instance segmentation, posture recognition and action recognition is used to achieve semantic perception of the physical disassembly space. The scene perception module includes an instance segmentation model, a posture recognition model and an action recognition model.

[0013] The human action behavior and scene information video in the power battery disassembly scene are simultaneously input into the instance segmentation model, the posture recognition model and the action recognition model, and the instance segmentation model, the posture recognition model and the action recognition model process the received video independently;

[0014] The instance segmentation model uses the Cascade R-CNN model to detect the location and outline of parts and tools;

[0015] The posture recognition model uses the ResNet model to detect the key points of the human skeleton. Since the movement of the lower body will not affect the action recognition in the actual battery disassembly, the key points of the human skeleton detected here are 15 upper body bone joints, and the ST-GCN model is used to process the key points of the human skeleton to obtain the human posture information;

[0016] The action recognition model compares the human posture information (i.e., the position information of the key points of the human skeleton) with the existing action data template according to the human posture information obtained by the posture recognition model, and determines the current disassembly process by identifying the operator's actions;

[0017] The processing results of the instance segmentation model, posture recognition model and action recognition model are input into the AR-assisted human-machine collaborative disassembly module. The AR-assisted human-machine collaborative disassembly module uses AR technology to simulate the human-machine collaborative disassembly process based on the received information, realizes the visualization of disassembly information and the judgment of human-machine safe distance, and ensures the safety of the disassembly system.

[0018] As the preferred technical solution:

[0019] As described above, in an AR assistance and scene perception system for human-machine collaborative power battery disassembly, human motion behaviors and scene information videos in the power battery disassembly scene are collected in real time by a depth camera.

[0020] As described above, in an AR-assisted and scene-aware system for human-machine collaborative power battery disassembly, the detection confidence of the ResNet model is set to 0.4±0.02, and redundant postures are eliminated through target refinement.

[0021] To avoid redundant gestures, i.e., multiple target boxes are recognized for one target, the gesture recognition detection confidence is usually set to a high value, but this will lead to the problem that the gesture estimator cannot detect people. The present invention sets the detection confidence of the ResNet model to a low value (0.4±0.02) to ensure a high detection recall rate. For the redundant detections generated by low confidence, target refinement is performed to eliminate redundancy.

[0022] As described above, in an AR-assisted and scene perception system for human-machine collaborative power battery disassembly, the AR-assisted human-machine collaborative disassembly module uses AR technology to simulate the human-machine collaborative disassembly process based on the information received. The specific process is: after obtaining the processing results of the instance segmentation model, posture recognition model and action recognition model in the scene perception module, the AR-assisted human-machine collaborative disassembly module simulates the upcoming process and visualizes the simulation process and results through AR glasses.

[0023] Among them, the AR-assisted human-machine collaborative disassembly module manages disassembly information through digital twin technology and assists operational decisions in real time; the AR-assisted human-machine collaborative disassembly module monitors the distance between the digital twin of the robotic arm and the human skeletal joints in real time to judge the safe distance and ensure the safety of the disassembly system.

[0024] The present invention provides an AR-assisted and scene-perception system for human-machine collaborative power battery disassembly, which realizes perception and auxiliary operation of the disassembly environment through a scene perception module and an AR-assisted human-machine collaborative disassembly module, improves disassembly efficiency and safety, and realizes efficient and safe human-machine collaborative power battery disassembly. The scene perception module determines the operator's process, position and posture, as well as the position and contour of the robot arm, power battery components and tools through instance segmentation, posture recognition and action recognition, to achieve semantic perception of the disassembly scene. The AR-assisted human-machine collaborative disassembly module uses AR technology to simulate the human-machine collaborative disassembly process, and realizes visualization of disassembly information and judgment of human-machine safety distance, to ensure the safety of the disassembly system. Experimental results show that the system can realize efficient and safe human-machine collaborative power battery disassembly.

[0025] The present invention has application value in many fields. For example, in the field of new energy vehicles, it can promote the development of power battery recycling technology, realize the efficient recycling of power batteries, and thereby reduce environmental pollution.

[0026] Beneficial effects:

[0027] (1) The AR-assisted and scene-perception system for human-machine collaborative power battery disassembly of the present invention uses a scene-perception module and an AR-assisted human-machine collaborative disassembly module to achieve efficient and safe human-machine collaborative power battery disassembly;

[0028] (2) The AR-assisted and scene-aware system for human-machine collaborative power battery disassembly of the present invention, in which the scene-aware module combines instance segmentation, posture recognition and action recognition methods to accurately determine the process performed by the operator, the position and posture of the operator, and the position and contour of the robot arm, power battery parts and tools, thereby improving the accuracy of scene perception of the disassembly environment;

[0029] (3) The present invention provides an AR-assisted and scene-perception system for human-machine collaborative power battery disassembly. The AR-assisted human-machine collaborative disassembly module simulates the human-machine collaborative disassembly process through AR technology, realizes the visualization of disassembly information and the judgment of human-machine safe distance, improves the safety of the disassembly system, reduces the cognitive burden of operators, and improves operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a framework diagram of the AR assistance and scene perception system for human-machine collaborative power battery disassembly;

[0031] Figure 2 A method for using a human-machine collaborative disassembly part based on augmented reality;

[0032] Figure 3 This is the AR human-machine collaboration sequence diagram;

[0033] Figure 4 This is a schematic diagram of AR human-machine collaboration;

[0034] Figure 5 This is a graph showing the experimental results comparing the efficiency of human-machine collaborative disassembly of power lithium batteries. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0036] An AR-assisted and scene-perception system for human-machine collaborative power battery disassembly, comprising a scene-perception module and an AR-assisted human-machine collaborative disassembly module;

[0037] The scene perception module includes an instance segmentation model, a posture recognition model, and an action recognition model;

[0038] The human action behavior and scene information video in the power battery disassembly scene are simultaneously input into the instance segmentation model, the posture recognition model and the action recognition model, and the instance segmentation model, the posture recognition model and the action recognition model process the received video independently respectively; wherein the human action behavior and scene information video in the power battery disassembly scene are collected in real time by the depth camera;

[0039] The instance segmentation model uses the Cascade R-CNN model to detect the location and outline of parts and tools;

[0040] The posture recognition model uses the ResNet model to detect the key points of the human skeleton. The detection confidence of the ResNet model is set to 0.4±0.02, and redundant postures are eliminated through target refinement. Since the movement of the lower body will not affect the action recognition in the actual battery disassembly, the key points of the human skeleton detected here are 15 upper body bone joints. The ST-GCN model is used to process the key point data of the human skeleton to obtain the human posture information.

[0041] The action recognition model determines the current disassembly process by identifying the operator's actions based on the human posture information obtained by the posture recognition model;

[0042] The processing results of the instance segmentation model, posture recognition model and action recognition model are input into the AR-assisted human-machine collaborative disassembly module. The AR-assisted human-machine collaborative disassembly module uses AR technology to simulate the human-machine collaborative disassembly process according to the received information, so as to realize the visualization of disassembly information and the judgment of human-machine safety distance. The specific process is as follows: after obtaining the processing results of the instance segmentation model, posture recognition model and action recognition model in the scene perception module, the AR-assisted human-machine collaborative disassembly module simulates the upcoming process and visualizes the simulation process and results through AR glasses.

[0043] The following uses a specific embodiment to illustrate an AR assistance and scene perception system for human-machine collaborative power battery removal of the present invention, as follows:

[0044] like Figure 1 As shown, the present invention aims at the problems of low efficiency and poor safety of manual disassembly of power batteries and the lack of adaptability and flexibility of robots to dynamic scenes, making it difficult to complete complex power battery disassembly tasks and ensure the safety of human-machine collaboration. Taking power battery disassembly as an example, an AR-assisted and scene-aware system for human-machine collaborative power battery disassembly is proposed. The system scene consists of four parts: 1) digital twin disassembly space, 2) AR human-machine disassembly, 3) physical disassembly space, and 4) scene perception; the details are as follows:

[0045] 1) Digital twin disassembly space: static data → modeling ← dynamic data;

[0046] Static data includes disassembly process information, robot information, 3D model information, and 2D drawing information;

[0047] Dynamic data includes human skeleton data, part tool position profile and disassembly status;

[0048] 2) AR human-machine disassembly, including registration / tracking, virtual-reality fusion, and human-machine interaction;

[0049] 3) After obtaining the processing results of the instance segmentation model, posture recognition model, and action recognition model in the scene perception module, the AR-assisted human-machine collaborative disassembly module simulates the upcoming process and visualizes the simulation process and results through AR glasses (Hololens).

[0050] The specific usage of the AR assistance and scene perception system for human-machine collaborative power battery removal is as follows:

[0051] First, a method combining instance segmentation, posture recognition and action recognition is used to achieve semantic perception of the physical disassembly space and obtain the current process and the operator's intention;

[0052] When the operator needs to pass the disassembly tool from the robot, the operator can convey relevant information with his body, such as stretching his hand. At this time, the posture recognition model recognizes the human skeleton, and then the action recognition model recognizes the human posture information by perceiving the continuous frames. The system extracts the local points of the human hand from the human skeleton to obtain the position difference between the key points and the robot arm fixture. The robot can place the tool on the human hand according to the position difference between the key points and the robot arm fixture. The scene perception part specifically includes the following steps:

[0053] (1) Collect human motion behavior and scene information videos in the power battery disassembly scene, obtain human posture information, the location of parts and tools, and color images and depth images of the disassembly process;

[0054] (1.1) Use a depth camera to shoot the dismantling personnel, collect human action behavior and scene information video, and obtain color images or depth images;

[0055] (1.2) Annotate the color image and depth image of the human action behavior and scene information video to obtain an .xml document containing the annotation information;

[0056] (1.3) Bind and save the .xml document containing the annotation information and the human body posture information, the positions of parts and tools, and the color images and depth images of the disassembly process in a txt file as a sample dataset for obtaining human body posture information, the positions of parts and tools, and the color images and depth images of the disassembly process;

[0057] (1.4) Preprocessing the human body posture information, the locations of parts and tools, and the color images and depth images of the disassembly process, including image denoising and data enhancement processing. The data enhancement processing adopts the Mosaic data enhancement method to expand the sample data set by randomly scaling, arranging, cropping and splicing the color images and depth images of the locations of parts and tools and the disassembly process that have been annotated;

[0058] (2) The Cascade R-CNN model is used to perform instance segmentation on human posture information, tool and component positions, and disassembly process color images to obtain the contour and position information of components, tools, and equipment. The CascadeR-CNN model is an improvement on the Faster R-CNN target detection model. The cascaded detectors effectively improve the accuracy of detecting coordinate information such as part position and contour.

[0059] A top-down approach is adopted to perform posture recognition on human body posture information, tool and component positions, and color images of disassembly processes through the ResNet model. Human body posture information is obtained by detecting the key points of the human skeleton. In actual battery disassembly, the movement of the lower body will not affect action recognition. The key points of the human skeleton detected here are 15 upper body bone joints.

[0060] (2.1) A top-down posture recognition method is used for human posture information, tool and component positions, and disassembly process color images. To avoid redundant postures, the posture recognition detection confidence is usually set to a high value, but the posture estimator may fail to detect the person. The present invention sets the detection confidence of the ResNet model to a low value (0.4±0.02) to ensure a high detection recall rate.

[0061] (2.2) For the redundant detections caused by low confidence, target refinement is performed to eliminate redundancy. The target box with the highest confidence is selected as a reference, and target box similarity detection is performed on some targets close to it, and the remaining target sets are eliminated until all redundant targets are eliminated.

[0062] Specifically, we first define the target frame similarity metric S(Bi, Bj|∧) to measure the target frame similarity, and define a threshold θ as the elimination criterion, where i, j are the human target frames with serial numbers, Bi, Bj are the human target frames with corresponding serial numbers identified, and ∧ is the parameter set of the function S(·). The elimination criterion in this paper can be written as follows:

[0063] f(Bi,Bj |∧, θ)=1[S(Bi,Bj|∧) <θ] (1);

[0064] The object box similarity is then measured. If S(·) is less than θ in the detection, the output of f(·) should be 1, indicating that Bi and Bj should be eliminated.

[0065] The similarity function Sbox(Bi,Bj) is defined as formula (2):

[0066] Sbox(Bi,Bj)=λΔ(xi,yi)+Δx+Δy (2);

[0067] Where xi, yi are the center points of the target frames, Δ(xi, yi) is the center coordinate deviation of the two target frames, Δx, Δy are the deviation values ​​of the width and height of the two target frames respectively, and λ is the weight parameter.

[0068] Given the detected redundant objects, the two parameters in the elimination criterion f(Bi, Bj|∧, θ) are optimized to achieve the maximum mean average precision (mAP) on the validation set.

[0069] During the testing process, one of the parameters is fixed and the other is optimized through iterations. Once convergence is achieved, the parameter will be fixed and will be used in the testing phase.

[0070] (3) Since the color image and the depth image are registered, there is a one-to-one correspondence between the pixels. Using the instance segmentation and posture recognition results, the two-dimensional data obtained from the instance segmentation and posture recognition are used to segment the depth information of the objects in the scene, and the data irrelevant to the object is removed, thereby obtaining the three-dimensional information of the object.

[0071] (4) The ST-GCN model is used to perform action recognition on the three-dimensional information of the segmented object, namely the key points of the human skeleton in continuous frames, so as to perceive the scene, obtain the human posture information, and thus obtain the current disassembly process. The ST-GCN model is designed based on the skeleton sequence, in which each node corresponds to a joint in the human body, allowing information to be integrated along the spatial and temporal dimensions.

[0072] In our experiments, the lower body is occluded by the workbench, and in the actual battery removal, the movement of the lower body does not affect the action recognition. Therefore, only 15 upper body skeleton joints are used as the input of the ST-GCN model to reduce the computational complexity and unnecessary trouble.

[0073] Spatial registration is the process of aligning spatial data or images from different sources into the same coordinate system or spatial frame. A transformation model is applied to align the data to the target coordinate system and perform error correction as needed.

[0074] Assuming that the coordinates of a point are (x, y), and the new coordinates after affine transformation are (x′, y′), the transformation relationship can be expressed as:

[0075]

[0076] in Control rotation, scaling and shearing, vector Controls translation.

[0077] 2. Refer to Figure 2 The human-machine collaborative disassembly part shown in the figure proposes an AR-assisted human-machine collaborative disassembly module to meet the safety and practicality requirements of human-machine collaborative power battery disassembly. It mainly includes four functions, as follows:

[0078] (I) Information visualization: visualize disassembly information;

[0079] (II) Digital twin disassembly: The human-machine collaborative disassembly process will be carried out in a virtual space with a 3D model; the simulation process is used to test the effect of HRC, while the synchronization process is used for real-time interaction;

[0080] (III) Control: After confirming the safety of the disassembly environment, humans can control the disassembly process;

[0081] (IV) Statistical analysis: will display and analyze the disassembly status, such as abnormal data, disassembly progress and other information.

[0082] Specifically, the robot arm position data obtained by the scene perception module will be transmitted to the human-machine collaboration module in real time to update the virtual robot arm position; the real-time virtual robot arm and the hand bone joint point position will be used to determine whether the safety distance is exceeded. If exceeded, the robot arm will be forced to stop and an alarm will be issued to the operator. At the same time, the AR-assisted human-machine collaborative disassembly module can visualize the disassembly information, including the original disassembly manual and other information collected by sensors in real time, thereby improving the operator's operating efficiency and reducing the training cycle for employees. Specifically, it includes the following steps:

[0083] (1) The system connects the scene perception and AR-assisted human-machine collaborative disassembly module, and can perform corresponding auxiliary operations based on the information of the next process node queried from the database, as well as the coordinate information of the human body position, posture, component position, contour, etc. taken in real time in the scene perception. When the confidence of the recognition result is low, the operator can be requested to intervene through the AR alarm to determine the current process; the content includes the process from starting collaboration, transferring tools, collaborative disassembly of the lid, and the determination of the safe distance (such as Figure 3 The interaction process between humans, AR and robots is shown in Figure 4 After the depth camera acquires the on-site disassembly scene video, it processes the video through the scene perception module to identify the operator's intention to need a screwdriver, and then drives the robot to grab the tool from TCP1 to TCP2; intention recognition is to judge the human body's operation intention by identifying human movements, and transmission drive is that the robot performs the corresponding operation after receiving the instruction. TCP refers to the Tool Center Point, which refers to the position of a specific point on the robot's end effector (such as welding head, spray gun, gripper, etc.). This point is usually the center point where the robot's operation task directly acts;

[0084] (2) The AR-assisted human-machine collaborative disassembly module defines the tasks that humans, robots, and AR need to perform in each process. For example, AR visualization (i.e., visualization of the disassembly process), text display of precautions, AR simulation prediction, disassembly videos, and animations will be played in a loop until the human-machine collaborative module feedback indicates that the last action of this process has been completed. Only then will the auxiliary information of the next process be displayed.

[0085] The processing results of the instance segmentation model, posture recognition model and action recognition model are input into the AR-assisted human-machine collaborative disassembly module. The AR-assisted human-machine collaborative disassembly module uses AR technology to simulate the human-machine collaborative disassembly process according to the received information, so as to realize the visualization of disassembly information and the judgment of human-machine safe distance.

[0086] like Figure 5 As shown, using an AR-assisted and scene perception system (AR-HRC system) for human-machine collaborative power battery disassembly of the present invention to perform power battery disassembly can improve disassembly efficiency, and the total disassembly time is reduced from 1157s of the preprogrammed method to 928s; in addition, the collision rate can also be reduced from 1.8% to 0.4%.

[0087] The pre-programmed method will cause the robot to run on redundant paths, and its motion trajectory, motion speed and motion start time are all pre-defined and cannot be changed, and the operator's operating efficiency is constantly changing, so the operator usually needs to wait for the robot operation to complete before proceeding to the next step. The AR-HRC system can obtain accurate position information of human bodies and objects and the current operation in real time through scene perception, so as to perform corresponding auxiliary operations in time and control the robot to run along the optimal route. Therefore, the working time of the robot and the waiting time of the operator will be shorter using the AR-HRC method. At the same time, AR visualization will also increase the work efficiency of the operator, so the operator of the AR-HRC method will work less time and the waiting time of the robot will be reduced accordingly.

Claims

1. An AR-assisted and scene-aware system for human-machine collaborative power battery removal, characterized in that: Including scene perception module and AR-assisted human-machine collaborative disassembly module; The scene perception module includes an instance segmentation model, a posture recognition model, and an action recognition model; The human action behavior and scene information video in the power battery disassembly scene are simultaneously input into the instance segmentation model, the posture recognition model and the action recognition model, and the instance segmentation model, the posture recognition model and the action recognition model process the received video independently; The instance segmentation model uses the Cascade R-CNN model to detect the location and outline of parts and tools; The posture recognition model uses the ResNet model to detect the key points of the human skeleton, and uses the ST-GCN model to process the key point data of the human skeleton to obtain the human posture information; The action recognition model determines the current disassembly process by identifying the operator's actions based on the human posture information obtained by the posture recognition model; The processing results of the instance segmentation model, posture recognition model and action recognition model are input into the AR-assisted human-machine collaborative disassembly module. The AR-assisted human-machine collaborative disassembly module simulates the human-machine collaborative disassembly process based on the received information using AR technology to realize the visualization of disassembly information and the judgment of the human-machine safe distance.

2. The AR-assisted and scene-aware system for human-machine collaborative power battery removal according to claim 1, characterized in that: The human motion behavior and scene information video in the power battery disassembly scenario are collected in real time by the depth camera.

3. The AR-assisted and scene-aware system for human-machine collaborative power battery removal according to claim 1, characterized in that: The detection confidence of the ResNet model is set to 0.4±0.02, and object refinement is performed to eliminate redundant poses.

4. The AR-assisted and scene-aware system for human-machine collaborative power battery removal according to claim 1, characterized in that: The AR-assisted human-machine collaborative disassembly module uses AR technology to simulate the human-machine collaborative disassembly process based on the received information. The specific process is: after obtaining the processing results of the instance segmentation model, posture recognition model and action recognition model in the scene perception module, the AR-assisted human-machine collaborative disassembly module simulates the upcoming process and visualizes the simulation process and results through AR glasses.

Images