Limb movement monitoring method and system based on visual information and electronic equipment

Abstract

The invention relates to the technical field of motion monitoring, and discloses a limb motion monitoring method and system based on visual information and electronic equipment, and the method comprises the steps: obtaining the visual information of a plurality of visual angles containing a target object; deducing limb movement information of the target object based on the visual information; based on a preset human body movement rule database, performing rationality verification on the limb movement information to obtain a rationality verification result of the current limb movement of the target object; and determining whether to trigger a danger judgment instruction based on the rationality verification result. According to the method, misjudgment which does not conform to the physical law of human motion and is caused by interference of visual information is effectively avoided, and the accuracy of motion mode recognition and the safety of the system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of motion monitoring technology, specifically to a method, system, and electronic device for limb motion monitoring based on visual information. Background Technology

[0002] In highly automated precision assembly workshops, advanced motion capture and recognition systems are typically deployed to ensure the safety of operators and collaborative robots. These systems monitor operator movements in real time using multi-view image sensors, instructing the robot to pause or adjust its trajectory if the operator's arm might enter a dangerous area. To address visual obstruction, the system integrates motion posture inference logic, using visible limbs, movement trends, and human biomechanics to complete the posture assessment and ensure accurate recognition.

[0003] However, in actual production, the introduction and installation of large, highly polished transparent protective covers with specific curvatures, coupled with workshop lighting illuminating their curved polished surfaces, produces strong and continuously changing specular reflections (glare). These glare beams have distinct geometric shapes (such as stripes or arcs), and their positions and forms can be predicted as the protective cover rotates, creating large areas of localized overexposure in the image sensor's view, dynamically "burning out" critical pixel information of the operator's arm within the camera's field of view.

[0004] Existing image preprocessing denoising and automatic exposure logic does not adequately consider this type of large-area, high-intensity, and rapidly moving dynamic structured glare. Traditional denoising optimizes for random noise or static local overexposure, but cannot suppress structured glare with specific shapes and motion trajectories. When balancing overall brightness in automatic exposure, compromises may be made due to extremely high brightness in glare areas, resulting in darkening of details in non-glare areas or failure to eliminate glare overexposure. Therefore, the system cannot distinguish between mixed images containing "real blurred arm outlines" and "false dynamic glare areas," mistakenly passing severely glare-contaminated image data as valid information to subsequent modules, creating potential problems in logical judgment.

[0005] When data severely contaminated by glare is fed into the motion and posture inference logic for the occluded area, the logic, which could have "filled in the gaps" in the occluded limbs based on incomplete visual information, mistakenly identifies the edges, shapes, and movement trajectories of the glare spots as limb features. This causes the "fused posture" output by the processing logic to be severely affected by the glare movement, generating a "ghost arm" whose position and orientation deviate from reality. It is an unreasonable combination of real arm information and glare pseudo-features, showing both blurred traces of the real limb and being dominated by the dynamic trajectory of the glare, presenting a movement pattern that does not conform to common sense of human biomechanics but seems "reasonable." For example, after the operator installs the protective cover, they need to put their hand into the safety recess inside the equipment to fasten the buckle (a "retraction" action away from the robot's danger zone). However, as the protective cover rotates, the glare on it sweeps across the danger zone that the robot is about to pass through. The system's mode classification logic misjudges the "ghost arm" movement dominated by the glare trajectory as the operator's arm "extending" towards the danger zone. The resulting technical dilemma is this: an advanced system designed to ensure safe human-machine collaboration, when the hardware and software are functioning normally and operators are operating in compliance with regulations, suffers a "blindness" and "deception" from dynamic glare caused by the introduction of a new component with a highly reflective curved surface. This leads to the core attitude inference logic being "blinded" and "deceived," creating a "hallucination" that continuously misjudges safe operations as dangerous behaviors. This not only frequently triggers unnecessary emergency stops for the robot, affecting production efficiency, but also undermines operators' trust in the system's safety judgments.

[0006] Therefore, in multi-angle motion capture pattern recognition systems, when an operator manipulates an object with a highly reflective surface, how to identify the high-risk perception state of "dynamic glare interference" by analyzing the inherent conflicts in multi-view data, such as strong local overexposure and inconsistencies between features and motion trajectories in biomechanics, and proactively reduce the confidence of the posture inference model or switch to a more conservative safety strategy, rather than confidently outputting an incorrect and arbitrary judgment of danger, is a technical problem that urgently needs to be solved. Summary of the Invention

[0007] This invention provides a method, system, and electronic device for limb movement monitoring based on visual information to solve the above-mentioned problems.

[0008] In a first aspect, the present invention provides a method for limb movement monitoring based on visual information, the method comprising: Obtain visual information from multiple perspectives of the target object; Based on visual information, infer the limb movement information of the target object; Based on a pre-set database of human movement patterns, the rationality of limb movement information is verified to obtain the rationality verification result of the target object's current limb movement. Based on the rationality verification results, determine whether to trigger a danger judgment instruction.

[0009] This invention combines the operator's limb movement information inferred from multi-angle visual information with the rationality verification of preset human movement physical laws. Thus, online, solely by analyzing the inherent conflicts in multi-view data, such as strong local overexposure and features and movement trajectories not conforming to biomechanics, it identifies the current high-risk perception state of "dynamic glare interference" and proactively reduces the confidence of its posture inference model or switches to a more conservative safety strategy, rather than confidently outputting an incorrect, arbitrary danger judgment. This achieves the goal of avoiding unnecessary safety alarms triggered by optical artifacts and ensuring true safety and efficiency in human-machine collaboration in complex optical environments.

[0010] In one optional implementation, inferring the limb movement information of the target object based on visual information includes: Obtain the first three-dimensional spatial information of the target object in the current working environment; Obtain the second and third-dimensional spatial information of the ambient light source in the current working environment; Acquire sensor information from the image sensor used to acquire visual information; Based on the first three-dimensional spatial information, the second three-dimensional spatial information, and sensor information, predict the area affected by light reflection in the image from each viewpoint; Based on the predicted region, a visual information reliability indicator is generated; Based on the reliability indicators of visual information, the weights of the visual information are adjusted. Based on the weighted visual information, the limb movement information of the target object is inferred.

[0011] In this embodiment, the system effectively avoids interference and errors caused by optical reflection when inferring limb movement information, making the inferred limb movement information closer to reality. This preprocessed and weighted limb movement information is then sent to the subsequent rationality verification step. Because the accuracy of the input information is improved, the efficiency and accuracy of the rationality verification are also enhanced, thereby avoiding erroneous judgments caused by visual information contamination and ensuring the reliability of the system's judgment of operator actions in complex optical environments.

[0012] In one optional implementation, based on first three-dimensional spatial information, second three-dimensional spatial information, and sensor information, the region in the image at each viewpoint affected by light reflection is predicted, including: Determine the geometric structure information of the target object's surface; Determine the local light reflection characteristics of the target object's surface; Based on geometric structure information, local light reflection characteristics, second three-dimensional spatial information, and sensor information, the reflection path of light on the surface of the target object is determined. Based on the direction and intensity of the reflected light, the areas in the image affected by light reflection are determined.

[0013] Through the more refined prediction methods described above, the system can generate more accurate visual information reliability indicators. When inferring operator limb movement information, this more accurate reliability indicator can be used to more precisely adjust the weights of visual information from different image regions, effectively reducing or ignoring visual information from regions with lower reliability. Therefore, even when the surface of the object being operated on has complex geometry and light reflection characteristics, and dynamic glare interference exists, the system can obtain more reliable visual input, thereby improving the accuracy of limb movement information inference. Ultimately, this allows the rationality verification step based on the physical laws of human movement to be based on data closer to reality, avoiding erroneous judgments caused by optical artifacts, thus improving the overall robustness and security of the multi-angle motion pattern recognition method.

[0014] In one alternative implementation, determining the local light reflection characteristics of the target object's surface includes: Obtain illumination information of the target object under different lighting conditions; Obtain observation information of the target object from different viewing angles; Based on illumination and observation information, determine the surface reflection behavior of the target object to light; Based on the reflection behavior, the local light reflection characteristics of the target object's surface are determined.

[0015] In this embodiment, when determining the reflection path of light on the surface of the object being operated on, physical simulation can be performed based on the actual reflection characteristics, thereby more accurately predicting the propagation direction and intensity of the reflected light.

[0016] In one optional implementation, when a portion of the surface of the target object is occluded during the acquisition of geometric information, determining the geometric information of the target object's surface includes: Identify the location of the obscured subject; Based on the location of the occluding subject and the acquired partial geometric information of the target object, the occluded area of ​​the target object is determined. Obtain historical geometric structure information of the target object at a previous time point; Obtain reference visual information from other perspectives; Based on reference visual information and historical geometric structure information, the geometric structure information of the occluded area is completed to obtain complete geometric structure information.

[0017] In this embodiment, by actively supplementing these missing fine geometric structural information, the integrity of the input data is ensured, thereby making the predicted optical reflection area more accurate and the reliability indication of the generated visual information more precise. This improves the effectiveness of weighting visual information from different image regions, ultimately enhancing the accuracy of operator limb movement inference and the reliability of rationality verification. This allows the system to continuously provide high-confidence action pattern recognition results even in complex and dynamically changing working environments, including partial occlusion, significantly enhancing the system's robustness and judgment accuracy.

[0018] In one optional implementation, based on a preset human movement pattern database, the rationality of limb movement information is verified to obtain the rationality verification result of the target object's current limb movement, including: Acquire scene feature information, which includes the identification information of the target object and / or the type information of the current task; Based on scene feature information, determine the limb movement parameters corresponding to the scene feature information; Based on limb movement parameters, the rationality of limb movement information is verified, and the rationality verification results are obtained.

[0019] This implementation method makes the verification of the rationality of limb movement information in motion monitoring no longer rigid and generalized, but adaptively adjustable according to individual differences of operators and specific requirements of the task. This significantly improves the accuracy of the rationality verification results, thereby making the system's judgment of operator actions more precise.

[0020] In one optional implementation, based on scene feature information, determining limb movement parameters corresponding to the scene feature information includes: Acquire motion performance data of the target object and information on the current execution stage of the task; Determine the initial limb movement parameters based on scene feature information; Based on motion performance data, determine the physiological state of the target object; Obtain a pre-established mapping table, which includes the mapping relationship between physiological state, execution stage information and limb movement parameter adjustment amount; Based on physiological state, execution stage information, initial limb movement parameters, and mapping relationship table, the adjustment amount of the initial limb movement parameters is calculated. Based on the adjustment amount, the initial limb movement parameters are adjusted to obtain the limb movement parameters.

[0021] This implementation no longer relies solely on static scene information to determine the rationality of limb movements, but takes into account the individual differences and real-time status of the operator, making the determined limb movement parameters more personalized and dynamically adaptable.

[0022] In one optional implementation, a mapping table is established through the following steps: Define conditional rules, which are used to map physiological state and execution stage information into specific adjustment amounts for limb movement parameters. Conditional rules define the parameter adjustment logic under different input combinations. Based on the conditional rules, physiological state and execution stage information are mapped to the adjustment amount of limb movement parameters, thereby establishing a mapping relationship between physiological state, execution stage information and limb movement parameter adjustment amount.

[0023] This implementation systematically maps the operator's physiological state and the stage of task execution into specific adjustments to limb movement parameters by defining conditional rules. This precise parameter adjustment mechanism avoids misjudgments or omissions caused by improper parameter settings, ensuring that the system can continuously provide reliable motion pattern recognition and safety assessments in complex and ever-changing human-machine collaborative environments.

[0024] Secondly, the present invention provides a limb movement monitoring system based on visual information, the system comprising: The acquisition module is used to acquire visual information from multiple perspectives of the target object; The inference module is used to infer the limb movement information of the target object based on visual information; The verification module is used to verify the rationality of limb movement information based on a preset human movement pattern database, and obtain the rationality verification result of the target object's current limb movement. The triggering module is used to determine whether to trigger a danger judgment instruction based on the rationality verification results.

[0025] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the visual information-based limb movement monitoring method of the first aspect or any corresponding embodiment described above.

[0026] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the visual information-based limb movement monitoring method described in the first aspect or any corresponding embodiment thereof.

[0027] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the visual information-based limb movement monitoring method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0028] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a schematic flowchart of a first method for monitoring limb movement based on visual information according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a limb movement monitoring system based on visual information according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0032] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0033] According to an embodiment of the present invention, a method for monitoring limb movement based on visual information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides a limb movement monitoring method based on visual information, which can be used on servers, terminals, and mobile terminals such as mobile phones and tablets. Figure 1 This is a flowchart of a limb movement monitoring method based on visual information according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain visual information containing multiple perspectives of the target object.

[0035] In this embodiment, the target object can be the operators in the workshop; the following description will also refer to the operators as the target object. Acquiring visual information from multiple perspectives of the operators refers to simultaneously capturing image data of the operators from different directions using multiple image sensors deployed around the work area. This image data can be continuous image data from different moments. Specifically, industrial-grade RGB-D cameras, stereo cameras, or multi-channel monocular camera arrays can be used, such as Intel RealSense D435i and Basler Ace series cameras. The main purpose is to acquire comprehensive, multi-dimensional raw perceptual data of the operators' movements, providing basic input for subsequent limb movement inference.

[0036] Step S102: Based on visual information, infer the limb movement information of the target object.

[0037] Based on this visual information, inferring the operator's limb movement information refers to estimating the position, posture, and changes over time of various parts of the operator's body in three-dimensional space through calculation and analysis, based on the acquired visual information from multiple perspectives. Specifically, this can be achieved using deep learning-based posture estimation algorithms, multi-view geometric triangulation, or skeleton tracking algorithms, such as posture recognition frameworks like OpenPose and AlphaPose. These frameworks primarily convert two-dimensional image data into three-dimensional motion state data that the system can use for action analysis and judgment.

[0038] Step S103: Based on a preset human movement pattern database, verify the rationality of limb movement information to obtain the rationality verification result of the target object's current limb movement.

[0039] A pre-defined human motion law database refers to a set of basic physical parameters and rules pre-stored within the system regarding the limits and characteristics of human limb movement. This may include joint angle limits, limb movement speed limits, limb movement acceleration limits, or movement smoothness constraints. Examples include the range of elbow flexion angles, the maximum linear velocity of the wrist, and the maximum angular velocity of each joint. Its primary purpose is to provide an objective and deterministic physical benchmark for verifying the authenticity and rationality of the limb movement information inferred by the visual system. Based on this, rationality verification refers to the process of comparing and checking the inferred limb movement information with the pre-defined human motion physical laws in real time to determine whether the movement information conforms to basic common sense of human physiology and biomechanics. This can be achieved using methods such as threshold comparison, range checking, or movement trajectory smoothness analysis. For example, calculating the inferred joint velocity and comparing it with the maximum permissible velocity is mainly used to identify and eliminate abnormal data that does not conform to the characteristics of real human movement, caused by optical artifacts or other non-physiological factors.

[0040] Step S104: Based on the rationality verification results, determine whether to trigger the danger judgment instruction.

[0041] The system corrects its judgment of the operator's actions based on the rationality verification results. That is, when the rationality verification results indicate that the limb movement information does not conform to the physical laws of human movement, the system actively adjusts or rejects the action judgment made based on the unreasonable information. This can be achieved by triggering a rejection signal, reducing the confidence level of the judgment, or switching to a conservative safety strategy. For example, when a "kinematic paradox" is detected, the system suppresses the issuance of a danger alarm. This is mainly to avoid triggering unnecessary safety responses that do not conform to the actual situation due to visual perception errors, thereby improving the accuracy and robustness of the system's judgment.

[0042] Based on the rationality verification results, the system's judgment of the operator's actions can be corrected, thereby preventing the system from issuing dangerous judgment commands when the rationality verification results indicate that the limb movement information does not conform to the physical laws of human movement.

[0043] This embodiment addresses the problem of erroneous motion judgments caused by optical artifacts in vision systems operating in complex optical environments. First, the system acquires visual information from multiple perspectives of the operator. This is the starting point of the entire motion recognition process; by capturing image data from different angles, the system obtains comprehensive perceptual input regarding the operator's movements. Based on this multi-perspective visual information, the system further infers the operator's limb movement information.

[0044] This step transforms the raw image data into quantifiable three-dimensional limb movement states, such as joint position, angle, and velocity, laying the foundation for subsequent motion analysis. However, the core of this method lies in its refusal to directly accept the limb movement information inferred by the visual system. Instead, it introduces a crucial verification step: based on pre-defined physical laws of human movement, it verifies the reasonableness of the inferred limb movement information and obtains a reasonableness verification result. This means that the system pre-stores a set of physical parameters and rules regarding the limits and characteristics of human limb movement, such as the range of motion of joints, maximum velocity, and acceleration of limb movement. The inferred limb movement information is compared with these physical laws in real time. If the inferred movement trajectory violates any pre-defined physical principle—for example, if a joint undergoes a displacement exceeding physiological limits in a very short time, or if the joint angle exceeds the normal range—then the movement information is deemed unreasonable. Ultimately, the system corrects its judgment of the operator's actions based on this reasonableness verification result.

[0045] Specifically, when the rationality verification result indicates that the limb movement information does not conform to the physical laws of human movement, the system will proactively avoid issuing a danger judgment command. This mechanism ensures that even if the visual system "sees" an unreal, seemingly dangerous "ghost arm" movement due to optical interference (such as glare), as long as this movement does not conform to the physical limits of human limbs, the system will not blindly trigger a safety alarm. It is precisely because of this independent verification and rejection mechanism based on common sense physics that this method can fundamentally avoid misjudgments caused by optical artifacts, thereby improving the robustness and accuracy of the system while ensuring the safety of human-machine collaboration.

[0046] In this embodiment, by combining the operator's limb movement information inferred from multi-angle visual information with the pre-set physical laws of human movement for rationality verification, the system identifies the current high-risk perception state of "dynamic glare interference" online, solely by analyzing the inherent conflicts in multi-view data, such as strong local overexposure and features and movement trajectories not conforming to biomechanics. It proactively reduces the confidence of the posture inference model or switches to a more conservative safety strategy, rather than confidently outputting an incorrect, arbitrary danger judgment. This avoids triggering unnecessary safety alarms due to optical artifacts and ensures the true safety and efficiency of human-machine collaboration in complex optical environments.

[0047] By introducing the physical laws of human movement to verify and correct the judgment of limb movement information, the problem of misjudgment caused by dynamic glare interference is effectively solved. It can verify the rationality of limb movement information by introducing the physical laws of human movement, and correct the system judgment based on the verification results. It can effectively avoid erroneous judgments that do not conform to the physical laws of human movement due to visual information interference (such as glare), and significantly improve the accuracy of action pattern recognition and the security of the system.

[0048] In some optional implementations, step S102 above, namely, inferring the limb movement information of the target object based on visual information, includes: Step S1021: Obtain the first three-dimensional spatial information of the target object in the current working environment.

[0049] The three-dimensional spatial information of the object being manipulated refers to the geometric shape, size, and optical properties of its surface material in three-dimensional space. It can be obtained through methods such as 3D scanning, CAD model import, or multi-view reconstruction.

[0050] Step S1022: Obtain the second three-dimensional spatial information of the ambient light source in the current working environment.

[0051] The three-dimensional spatial information of ambient light sources refers to the location, type, and intensity distribution of all lighting sources in the working environment in three-dimensional space. This information can be obtained through manual measurement, capture by a light field camera, or by pre-set models.

[0052] Step S1023: Obtain sensor information of the image sensor used to acquire visual information.

[0053] The sensor information of an image sensor refers to the inherent properties of the image sensor itself and its position and orientation in three-dimensional space. Specifically, it includes internal parameters such as focal length, principal point, and distortion coefficient, as well as external parameters such as the rotation and translation of the camera relative to the world coordinate system. It can be obtained through camera calibration or preset configuration.

[0054] Step S1024: Based on the first three-dimensional spatial information, the second three-dimensional spatial information, and the sensor information, predict the area affected by light reflection in the image from each viewpoint.

[0055] Predicting the regions in an image affected by optical reflection refers to determining which pixel regions in an image may suffer from visual information distortion or saturation due to optical reflection by simulating the reflection process of light on the surface of the object being manipulated. This can be achieved using ray tracing algorithms, physically based rendering models, or empirical reflection models, with the aim of identifying potentially interfering regions in the image.

[0056] Step S1025: Based on the predicted region, generate a visual information reliability indicator.

[0057] Visual information reliability indicators are identifiers used to quantify the credibility of visual information in different regions of an image. Specifically, they can be a two-dimensional matrix or mask, in which each pixel or region is assigned a reliability score or binary label, with the aim of providing guidance for subsequent visual information processing.

[0058] Step S1026: Based on the visual information reliability indicator, adjust the weight of the visual information to reduce or ignore visual information from areas with lower reliability.

[0059] Weighting refers to assigning different levels of importance to visual information from different image regions based on their reliability when fusing them. This can be achieved through multiplication factors, weighted averaging, or selective ignoring, with the aim of reducing or eliminating the negative impact of unreliable visual information on the overall judgment.

[0060] Step S1027: Based on the weighted visual information, infer the limb movement information of the target object.

[0061] In this embodiment, the accuracy of limb movement information inference is improved by introducing a mechanism for evaluating and adjusting the reliability of visual information before verifying the reasonableness of limb movement information. Specifically, the system first acquires the three-dimensional spatial information of the object being manipulated and the three-dimensional spatial information of the ambient light source. This information is the key foundation for constructing a real-world optical environment model. Based on this, and combined with the parameters of the image sensor, the system can accurately predict which areas in the image will be affected by optical reflection. This prediction process utilizes the physical principles of light propagation and reflection, simulating the path of light starting from the light source, being reflected from the surface of the object being manipulated, and finally entering the image sensor, thereby locating the affected areas on the image plane. Subsequently, based on these predicted affected areas, the system generates a visual information reliability indicator, clearly identifying which areas in the image may have unreliable visual information. When inferring the operator's limb movement information based on visual information from multiple perspectives, the system no longer blindly accepts all visual data, but adjusts the weight of visual information from different image regions according to the visual information reliability indicator. This means that visual information from areas with lower reliability will have its importance reduced, or even be completely ignored, while visual information from areas with higher reliability will be given higher weight. In this way, the system can effectively avoid interference and errors caused by optical reflection when inferring limb movement information, making the inferred limb movement information closer to reality. This preprocessed and weighted limb movement information is then sent to the subsequent rationality verification step. Because the accuracy of the input information is improved, the efficiency and accuracy of rationality verification are also enhanced, thus avoiding erroneous judgments caused by visual information contamination and ensuring the reliability of the system's judgment of operator actions in complex optical environments.

[0062] In some optional implementations, step S1024 above, namely predicting the region affected by light reflection in the image from each viewpoint based on the first three-dimensional spatial information, the second three-dimensional spatial information, and the sensor information, includes: Step a1: Determine the geometric structure information of the target object's surface.

[0063] The geometric structure information of the object's surface refers to acquiring more detailed and refined geometric features, such as its microscopic or local structural information like its concavity, curvature, and texture, rather than just its overall three-dimensional outline. This can be achieved through data acquisition using a high-resolution 3D scanner or through structured light projection combined with multi-view reconstruction technology. The purpose is to provide fundamental data for the accurate simulation of light reflection on the object's surface.

[0064] Step a2: Determine the local light reflection characteristics of the target object's surface.

[0065] The local light reflection characteristics of an object's surface refer to determining the reflection behavior of different areas of the object's surface to incident light, including but not limited to the proportion of diffuse reflection, specular reflection, or mixed reflection, as well as optical properties such as reflection intensity and color. This can be achieved by establishing a two-way reflectance distribution function (BRDF) model by measuring the intensity of reflected light at different incident and viewing angles, or by inferring it through analyzing the physical properties of the surface material. The aim is to more realistically simulate the intensity and direction of light reflection on an object's surface, thereby accurately predicting the areas in an image affected by light reflection.

[0066] Step a3: Based on geometric structure information, local light reflection characteristics, second three-dimensional spatial information, and sensor information, determine the reflection path of light on the surface of the target object.

[0067] This process comprehensively considers various factors such as the geometry of the object's surface, its reflectivity, the location of the light source, and camera parameters. It simulates the process of light emanating from the light source, being reflected off the object's surface, and finally reaching the image sensor using ray tracing or similar methods. By determining the light reflection path, it's possible to accurately identify which areas of light will reflect back to the camera, as well as the intensity and direction of the reflected light.

[0068] Step a4: Based on the propagation direction and intensity of the reflected light, determine the area in the image affected by light reflection.

[0069] This embodiment improves the accuracy of visual information reliability assessment by performing more refined processing on regions affected by optical reflection in the predicted image. Specifically, when predicting regions affected by optical reflection in the image, the fine geometric structure information of the object's surface is first acquired. This is because, unlike considering only the overall three-dimensional spatial information of the object, fine geometric structure information can reflect details such as the surface's concavity and curvature, which significantly affect the reflection path of light. By acquiring this fine structure information, the reflection behavior of light on the surface can be simulated more accurately, providing a precise basis for subsequent reflection path determination. Based on this, the local light reflection characteristics of the object's surface are acquired. This is because different materials and surface treatments have different light reflection characteristics, such as diffuse reflection or specular reflection. Even with the same material, the reflection characteristics of different regions may differ. By acquiring the local light reflection characteristics of the object's surface, the intensity and direction of light reflection on the surface can be simulated more realistically. Furthermore, based on the acquired fine geometric structure information, local light reflection characteristics, and existing three-dimensional spatial information of the ambient light source and image sensor parameters, the reflection path of light on the object's surface is determined. This process comprehensively considers various factors such as the geometry of the object's surface, its reflectivity, the location of the light source, and camera parameters. Using ray tracing or similar methods, it simulates the process of light emanating from the light source, reflecting off the object's surface, and finally reaching the image sensor. By determining the light reflection path, it's possible to accurately identify which areas will reflect light to the camera, as well as the intensity and direction of the reflected light. Finally, based on the propagation direction and intensity of the reflected light, the regions in the image affected by light reflection are determined. This step translates the calculated light reflection path into specific regions on the image. By analyzing the propagation direction and intensity of the reflected light, it's possible to determine which pixels will be affected by the reflected light and the degree of impact. For example, areas with high reflected light intensity may be overexposed, while areas with low reflected light intensity may appear shadowed.

[0070] Through the more refined prediction methods described above, the system can generate more accurate visual information reliability indicators. When inferring operator limb movement information, this more accurate reliability indicator can be used to more precisely adjust the weights of visual information from different image regions, effectively reducing or ignoring visual information from regions with lower reliability. Therefore, even when the surface of the object being operated on has complex geometry and light reflection characteristics, and dynamic glare interference exists, the system can obtain more reliable visual input, thereby improving the accuracy of limb movement information inference. Ultimately, this allows the rationality verification step based on the physical laws of human movement to be based on data closer to reality, avoiding erroneous judgments caused by optical artifacts, thus improving the overall robustness and security of the multi-angle motion pattern recognition method.

[0071] In some optional implementations, step a2 above, namely determining the local light reflection characteristics of the target object surface, includes: Step b1: Obtain the illumination information of the target object under different lighting conditions.

[0072] Step b2: Obtain observation information of the target object from different viewing angles.

[0073] Step b3: Based on the illumination information and observation information, determine the light reflection behavior of the target object's surface.

[0074] Step b4: Based on the reflection performance, determine the local light reflection characteristics of the target object's surface.

[0075] Reflection performance refers to the way and intensity distribution of light reflected from the surface of an object under specific lighting and observation conditions. Specifically, it can be reflected by analyzing changes in highlights, shadows, color saturation, and texture details in an image. Its purpose is to quantify the response pattern of the object's surface to incident light. Local light reflection characteristics refer to the light reflection properties of the object's surface at the microscale. Specifically, it describes the influence of physical properties such as material, roughness, and optical coatings on light reflection. Its purpose is to establish a light reflection model of the object's surface to provide a basis for accurately predicting the light reflection path.

[0076] This embodiment acquires visual information of the object under different lighting conditions and from different viewing angles, enabling it to comprehensively capture the reflection of the object's surface under various actual lighting and viewing angles. Based on this multi-dimensional, multi-condition visual data, the system can comprehensively analyze and determine the object's surface reflection characteristics, such as identifying the shape, brightness, and positional variation patterns of highlight areas, as well as the formation and dissipation patterns of shadows. It is precisely this accurate capture and analysis of these reflection characteristics that allows the system to further infer the local light reflection characteristics of the object's surface, such as whether its material exhibits specular or diffuse reflection, and its surface micro-roughness and other physical properties. This accurate acquisition of local light reflection characteristics provides crucial input for subsequent motion pattern recognition processes.

[0077] Specifically, when determining the reflection path of light on the surface of the object being manipulated, physical simulations can be performed based on the actual reflection characteristics, thereby more accurately predicting the propagation direction and intensity of the reflected light. This directly affects the identification of areas in the image affected by light reflection, ensuring that the predicted glare or shadow areas match the actual situation. Furthermore, when the system generates a visual information reliability indicator based on these predicted areas, this indicator accurately reflects which areas in the image may be affected by optical reflection. This allows for effective weighting of visual information from different image regions when inferring limb movement information, reducing or ignoring visual information from less reliable regions. Ultimately, this enables the system to infer limb movement information that closely approximates the operator's actual movements, avoiding judgment errors caused by optical reflection artifacts, thus improving the accuracy of motion pattern recognition and ensuring the safety of human-machine collaboration.

[0078] In some optional implementations, when a portion of the surface of the target object is occluded during the acquisition of geometric information, determining the geometric information of the target object's surface includes: Step c1: Identify the location of the occluded subject.

[0079] The occlusion subject refers to an entity that obstructs the surface of the object being operated on within the field of view of the image sensor. It can be a limb of the operator, such as an arm or a hand, or a tool used during the operation, such as a wrench or a clamp. The purpose is to identify the source of the occlusion and provide a basis for subsequently determining the occlusion area.

[0080] Step c2: Based on the location of the occluding subject and the acquired partial geometric information of the target object, determine the occluded area of ​​the target object.

[0081] The acquired partial geometric information of the operation object refers to the high-precision geometric data of the operation object surface that has been successfully collected by technologies such as 3D scanning, structured light projection, or multi-view reconstruction before the occlusion occurs or in the unoccluded area. Its purpose is to provide known and reliable geometric references so as to accurately define the occlusion range by combining them with the position of the occluded subject.

[0082] The occluded area of ​​the object being manipulated refers to the physical area on the surface of the object being manipulated that is completely or partially occluded by the occluded subject under the view of a specific image sensor, making it impossible to directly obtain its fine geometric structure information. Its purpose is to accurately define the target area that needs to be supplemented with information and improve the pertinence of the supplementation.

[0083] Step c3: Obtain the historical geometric structure information of the target object at previous moments.

[0084] The geometric structure information of the previous moment refers to the complete or partial detailed geometric structure data of the operation object that the system has successfully acquired and stored before the current moment. Its purpose is to take advantage of the fact that the geometric structure of the operation object usually remains relatively stable in a short period of time, and use it as an effective reference for the currently occluded area to assist in information completion.

[0085] Step c4: Obtain reference visual information from other perspectives.

[0086] Reference visual information from other perspectives refers to image acquisition devices deployed in different locations or with different perspectives, in addition to the currently occluded image sensor. For example, these could be other cameras located above, to the side, or behind the work area. Their purpose is to provide visual data captured from different angles to obtain information about the area occluded by the current perspective.

[0087] Step c5: Based on reference visual information and historical geometric structure information, complete the geometric structure information of the occluded area to obtain complete geometric structure information.

[0088] Completing the geometric structure information of the occluded area refers to using algorithms or models to reconstruct or infer the high-precision geometric data missing from the surface of the object due to occlusion, using existing multi-source data, such as visual data from other image sensors and geometric structure information from previous moments. The purpose is to restore the complete geometric structure of the object and provide a complete and reliable data foundation for subsequent action pattern recognition.

[0089] Specifically, when the system acquires detailed geometric information about the surface of the object being manipulated, it first identifies the location of the occluding subject. This identification forms the basis for accurately defining the occluded area, enabling the system to initially determine which parts might be occluded. Based on this, the system combines the acquired detailed geometric information of the object to precisely determine the occluded area. This combination utilizes existing reliable data, avoids misjudging unoccluded areas, and ensures the targeted nature of the information completion. To effectively complete missing geometric information, this embodiment further acquires visual data from other image sensors. Since occlusion is usually relative to a specific viewpoint, image sensors from other viewpoints can capture information about areas occluded by the current viewpoint, thus providing new and complementary data sources. Simultaneously, the system also acquires the geometric information of the object at a previous moment. Considering that the geometric structure of the object usually remains relatively stable in a short period, data from previous moments can serve as a valid reference for the currently occluded area, helping to maintain the continuity and rationality of the completed geometric structure. Ultimately, the system comprehensively utilizes visual data from other image sensors and the geometric structure information of the manipulated object at previous moments to complete the fine geometric structure information of the occluded area. This multi-source information fusion method enables the system to reconstruct the geometric details of the occluded area more comprehensively and accurately, thereby overcoming the problem of incomplete visual information caused by occlusion.

[0090] By supplementing the aforementioned detailed geometric information, a more complete and reliable data foundation is provided for the subsequent action pattern recognition process. In the entire motion monitoring method, the detailed geometric information of the object's surface is a key input for predicting the region affected by optical reflection in the image. When this geometric information is missing due to occlusion, the accuracy of subsequent optical reflection prediction, visual information reliability indicator generation, and limb movement information inference will be affected. This solution actively supplements this missing detailed geometric information, ensuring the integrity of the input data, thereby making the predicted optical reflection region more accurate and the generated visual information reliability indicator more precise. This improves the effectiveness of weighting visual information from different image regions, ultimately enhancing the accuracy of operator limb movement information inference and the reliability of rationality verification. This allows the system to continuously provide high-confidence action pattern recognition results even in complex and dynamically changing working environments, even with partial occlusion, significantly enhancing the system's robustness and judgment accuracy.

[0091] In some optional implementations, step S103 above, namely, verifying the rationality of limb movement information based on a preset human movement pattern database to obtain the rationality verification result of the target object's current limb movement, includes: Step d1: Obtain scene feature information, which includes the identification information of the target object and / or the type information of the current task.

[0092] Scene feature information refers to specific data related to the current operating environment or operators, which can be obtained through sensors, database queries, or manual input. Operator identification information refers to unique identifiers used to distinguish different operators, which can be obtained through biometric data, employee ID, RFID tag information, or pre-set user profiles. Task type information refers to the classification or description data of the currently executed task, which can be obtained through task ID, task description text, pre-set task templates, or visual identification of the current task object.

[0093] Step d2: Based on the scene feature information, determine the limb motion parameters corresponding to the scene feature information.

[0094] Limb motion parameters refer to a set of values ​​or rules used to define the reasonable range of human limb movement. They can be defined by means of maximum joint angle, maximum linear velocity, maximum angular velocity, maximum acceleration, motion smoothness threshold, or biomechanical constraints of specific movement patterns.

[0095] Step d3: Based on the limb movement parameters, verify the rationality of the limb movement information and obtain the rationality verification result.

[0096] This embodiment dynamically optimizes the verification process of the physical laws of human movement by introducing the perception and utilization of scene feature information. Specifically, after the system acquires visual information from multiple perspectives of the operator and infers the operator's limb movement information, before verifying the rationality of the limb movement information, the system first acquires the current scene feature information. This scene feature information may include the operator's identification information, such as the operator's identity, height, weight, age, or individual difference data such as their historical movement habits; it may also include information about the type of task, such as whether the current task requires fine operation, large-amplitude movement, or heavy-duty handling.

[0097] Based on the acquired scene feature information, the system intelligently determines the limb movement parameters that match the current scene features. This means that the system no longer uses a fixed set of universal human movement physics laws to verify limb movements in all situations, but can dynamically adjust the verification standards according to the characteristics of the specific operator or task. For example, for a tall operator, whose range of motion may be relatively large, the system can correspondingly relax the restrictions on their range of motion; for a task requiring precise operation, the system can more strictly limit the speed and acceleration of limb movements to ensure the accuracy of the action.

[0098] Subsequently, the system uses these dynamically determined limb movement parameters to verify the rationality of the previously inferred limb movement information, thereby obtaining a more accurate rationality verification result. This mechanism of dynamically adjusting verification parameters allows the rationality verification process to better adapt to the complexity and diversity of human movement in real-world applications, avoiding inaccurate verification problems caused by using fixed parameters.

[0099] In this way, the rationality verification of limb movement information in motion monitoring is no longer rigid and one-size-fits-all, but can adaptively adjust according to individual differences of operators and specific requirements of the task. This significantly improves the accuracy of rationality verification results, thereby making the system's judgment of operator actions more precise. When the rationality verification result indicates that the limb movement information does not conform to the physical laws of human movement, the system can more reliably avoid issuing dangerous judgment commands, especially when visual information may be interfered with and produce artifacts such as "ghost arms." This dynamically adjusted verification mechanism can more effectively identify abnormal movements that do not conform to real physiological laws, thereby avoiding unnecessary robot abrupt stops and ensuring the efficiency and safety of human-machine collaboration. This refined management of verification standards enables the entire system to provide more reliable and intelligent safety judgments in complex and ever-changing working environments.

[0100] In some optional implementations, step d2 above, namely determining the limb motion parameters corresponding to the scene feature information based on the scene feature information, includes: Step e1: Obtain motion performance data of the target object and information on the current execution stage of the task.

[0101] Motor performance data refers to data that quantifies the limb movement characteristics of operators during task execution. It may include, but is not limited to, indicators such as the operator's average movement speed, range of motion, smoothness of movement, reaction time, and repeatability or variability of specific movements. Its purpose is to provide basic information for assessing the operator's movement habits and abilities.

[0102] The execution phase information of a task refers to information describing the current progress of the task. It can include the start phase, intermediate execution phase, end phase, or completion status of a specific subtask. It can be obtained based on the preset nodes of the task flow, timestamps, or the interaction status between the operator and the equipment. Its purpose is to reflect the impact of physiological or psychological changes that may occur in the operator at different task stages on limb movement.

[0103] Step e2: Determine the initial limb movement parameters based on scene feature information.

[0104] Initial limb movement parameters refer to the reasonable range or threshold of limb movement determined solely based on scene feature information (e.g., job type, environmental layout) without considering the operator's real-time physiological state and task execution stage. These parameters may include the upper limit of limb movement speed, the upper limit of acceleration, the range of joint angles, and the tolerance for trajectory deviation of specific actions. The purpose is to provide a scene-based universal reference benchmark.

[0105] Step e3: Based on the motion performance data, determine the physiological state of the target object.

[0106] Physiological state refers to the current physical and mental condition of the operator, which may include fatigue level, attention level, concentration or stress response. It can be judged based on various physiological indicators such as motor performance data (e.g., slow movement, increased tremors), heart rate, eye tracking data or EEG signals. Its purpose is to assess whether the operator is currently in a state that affects their motor ability.

[0107] Step e4: Obtain the pre-established mapping relationship table, which includes the mapping relationship between physiological state, execution stage information and limb movement parameter adjustment amount.

[0108] Establishing a mapping relationship between physiological state, execution stage information, and limb movement parameter adjustments involves constructing an association model or lookup table to link the operator's physiological state (e.g., fatigue level, mental concentration) and the execution stage of the task (e.g., task start, core operation, task end) with specific adjustment values ​​or proportions of limb movement parameters (e.g., speed, acceleration, joint angle range). This can be achieved using machine learning models, expert experience systems, fuzzy logic rule bases, etc., with the aim of providing quantifiable and reliable criteria for subsequent parameter adjustments. The limb movement parameter adjustment refers to one or a set of values ​​used to incrementally or proportionally modify the initially determined limb movement parameters. This adjustment can be an absolute value or a relative proportion, aiming to make the initial parameters more accurately reflect the operator's actual movement ability and characteristics under specific physiological states and task stages.

[0109] Step e5: Based on physiological state, execution stage information, initial limb movement parameters, and mapping relationship table, calculate the adjustment amount of the initial limb movement parameters.

[0110] During actual operation, the system acquires real-time information on the operator's physiological state and the current stage of the task, combined with initial limb movement parameters determined based on scenario characteristics. Based on a pre-established mapping relationship, the system can accurately calculate the necessary adjustments to the limb movement parameters for the current situation.

[0111] Step e6: Based on the adjustment amount, adjust the initial limb motion parameters to obtain the limb motion parameters.

[0112] The final limb movement parameters refer to the rationality judgment criteria for limb movement, obtained through dynamic correction based on the initial limb movement parameters and comprehensively considering the operator's physiological state and the stage of task execution. This aims to improve the accuracy and personalization of limb movement rationality verification. Adjusting the initial limb movement parameters refers to modifying or optimizing them according to specific logic or models. This may include amplifying, reducing, shifting, or redefining the range of the parameters, with the goal of adapting the parameters to the operator's real-time changes, thereby more accurately reflecting the rationality of their current movement.

[0113] This embodiment refines the process of determining limb movement parameters by incorporating considerations of the operator's real-time status and task progress. First, the system acquires the operator's motion performance data, reflecting their movement characteristics during actual operation, such as the speed, amplitude, and stability of their movements. Simultaneously, the system acquires information on the execution stage of the task, enabling it to understand the specific phase of the task. This is because the operator's physiological and psychological state may change at different stages of the task, thus affecting their limb movement patterns. Based on this, the system determines a set of initial limb movement parameters according to pre-defined scenario feature information. These parameters are based on general settings specific to the work scenario, providing a foundation for subsequent adjustments.

[0114] Subsequently, the system assesses the operator's physiological state based on the acquired motion performance data. For example, if motion performance data shows sluggish movements or irregular shaking, the system can determine that the operator may be fatigued. Furthermore, the system establishes a correlation model to predefine or learn the specific adjustments that limb movement parameters may require under different physiological states and at different stages of task execution. This mapping relationship is the foundation for precise adjustments, ensuring the quantification and predictability of the adjustment process. During actual operation, the system acquires the operator's physiological state and the current task execution stage information in real time, combined with the initial limb movement parameters determined based on scenario characteristics. Based on the pre-established mapping relationship, the system can accurately calculate the required adjustment amount of limb movement parameters for the current situation. This calculation process considers individual differences among operators and dynamic changes in the task, making the adjustments more personalized and refined. Finally, the system comprehensively considers the assessed physiological state, the acquired execution stage information, and the initial limb movement parameters, dynamically adjusting the initial parameters to obtain the final limb movement parameters. For example, when the system determines that an operator is fatigued or the task is in its later stages, it can appropriately tighten or loosen the thresholds of certain limb movement parameters to better reflect the operator's current actual movement ability and task requirements. These final parameters can more accurately reflect the operator's true movement ability and behavioral patterns in a specific situation.

[0115] The aforementioned adjustment process provides a quantifiable and reliable adjustment mechanism. This allows the limb movement parameters used in subsequent kinematic rationality verification to more closely reflect the actual situation of the operator. Even when optical interference leads to inaccurate visual information, verification can be based on more reliable parameters. For example, when an operator is fatigued, their limb movement limits and acceleration decrease, and joint range of motion may be limited. Through the adjustments in this scheme, these parameters are tightened accordingly, enabling the kinematic verification module to more sensitively identify "ghost arm" trajectories caused by optical artifacts that exceed the actual human movement limits under fatigue, thereby avoiding misjudgments. This refined adjustment of limb movement parameters allows the entire motion pattern recognition system to more accurately determine the operator's true intentions and actions in complex and changing working environments, significantly improving the accuracy and reliability of safety judgments.

[0116] In this way, the solution no longer relies solely on static scene information to determine the rationality of limb movements, but incorporates individual differences and real-time states of the operator, making the determined limb movement parameters more personalized and dynamically adaptable. This allows subsequent rationality verification of limb movement information based on these parameters to more accurately identify genuine abnormal movements, rather than normal deviations caused by changes in operator state or task phase. Furthermore, this improves the overall accuracy of the multi-angle motion pattern recognition method, effectively avoiding misjudgments caused by parameter mismatches. Thus, while ensuring the safety of human-machine collaboration, it reduces unnecessary system intervention and improves production efficiency and system reliability.

[0117] In some optional implementations, a mapping table is established through the following steps: Define conditional rules, which are used to map physiological state and execution stage information into specific adjustment amounts for limb movement parameters. Conditional rules define the parameter adjustment logic under different input combinations. Based on the conditional rules, physiological state and execution stage information are mapped to the adjustment amount of limb movement parameters, thereby establishing a mapping relationship between physiological state, execution stage information and limb movement parameter adjustment amount.

[0118] Conditional rules refer to a set of pre-defined logical judgments or functional relationships used to guide the system in determining the output result based on specific input conditions. They can be implemented using decision trees, rule engines, expert systems, or lookup table-based logic, aiming to provide a structured basis for the conversion of physiological state and execution phase information into limb movement parameter adjustments. Physiological state refers to the comprehensive reflection of the operator's current physical and mental state. It can include indicators such as fatigue level, concentration, heart rate, respiratory rate, body temperature, or muscle activity electrical signals, aiming to reflect the operator's actual physical condition at different times or under different task loads. Execution phase information refers to the specific stage or progress of the task within the entire process. It can include the task's initiation phase, core operation phase, completion phase, or exception handling phase, aiming to reflect the task's context, as different stages may have different requirements for the operator's actions and risk levels. Limb movement parameter adjustments refer to the specific values ​​or proportions used to correct initial limb movement parameters. This can include increments or decrements in requirements for joint angle range, upper limit of movement speed, acceleration threshold, or movement smoothness, with the aim of making limb movement parameters more closely match the operator's actual state and task needs. A mapping relationship refers to the correspondence established between a set of input values ​​and a set of output values. This can be implemented using lookup tables, multidimensional arrays, mathematical function models, or machine learning models, with the aim of transforming complex physiological and task scenarios into specific parameter adjustment instructions.

[0119] This embodiment systematically maps the operator's physiological state and the execution stage of the task to specific adjustment amounts for limb movement parameters by defining conditional rules. These conditional rules clarify the logic for adjusting limb movement parameters under different combinations of physiological states and execution stages. Based on this, the system, according to these preset conditional rules, maps real-time physiological state information and execution stage information to corresponding limb movement parameter adjustment amounts, thereby establishing a clear and operable mapping relationship. This mapping relationship allows subsequent adjustments to initial limb movement parameters to move beyond simple fixed values ​​or rough judgments, enabling dynamic and precise calculation of adjustment amounts based on the operator's actual physical condition and the specific stage of the task. For example, when the operator is fatigued and the task enters a fine-tuning stage, the system can calculate a specific adjustment amount based on the conditional rules, making the validity verification range of limb movement parameters more rigorous and thus identifying potential abnormal movements earlier. It is precisely this conditional rule-based mapping relationship that gives the adjustment process of initial limb movement parameters higher accuracy and adaptability. This further enhances the accuracy of determining limb movement parameters based on scene feature information, enabling a more accurate assessment of whether the operator's movements conform to the physical laws of human movement when verifying the rationality of limb movement information. This precise parameter adjustment mechanism avoids misjudgments or omissions caused by improper parameter settings, ensuring that the system can continuously provide reliable motion pattern recognition and safety judgments in complex and ever-changing human-machine collaborative environments.

[0120] This embodiment also provides a limb movement monitoring system based on visual information, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0121] This embodiment provides a limb movement monitoring system based on visual information, such as Figure 2 As shown, the system includes: The acquisition module 201 is used to acquire visual information containing multiple perspectives of the target object; Inference module 202 is used to infer the limb movement information of the target object based on visual information; The verification module 203 is used to verify the rationality of limb movement information based on a preset human movement pattern database, and obtain the rationality verification result of the target object's current limb movement. Trigger module 204 is used to determine whether to trigger a danger judgment instruction based on the rationality verification result.

[0122] In some alternative implementations, the inference module 202 is specifically used for: Obtain the first three-dimensional spatial information of the target object in the current working environment; Obtain the second and third-dimensional spatial information of the ambient light source in the current working environment; Acquire sensor information from the image sensor used to acquire visual information; Based on the first three-dimensional spatial information, the second three-dimensional spatial information, and sensor information, predict the area affected by light reflection in the image from each viewpoint; Based on the predicted region, a visual information reliability indicator is generated; Based on the reliability indicators of visual information, the weights of the visual information are adjusted. Based on the weighted visual information, the limb movement information of the target object is inferred.

[0123] In some alternative implementations, the inference module 202 is further specifically used for: Obtain illumination information of the target object under different lighting conditions; Obtain observation information of the target object from different viewing angles; Based on illumination and observation information, determine the surface reflection behavior of the target object to light; Based on the reflection behavior, the local light reflection characteristics of the target object's surface are determined.

[0124] In some alternative implementations, the inference module 202 is further specifically used for: Identify the location of the obscured subject; Based on the location of the occluding subject and the acquired partial geometric information of the target object, the occluded area of ​​the target object is determined. Obtain historical geometric structure information of the target object at a previous time point; Obtain reference visual information from other perspectives; Based on reference visual information and historical geometric structure information, the geometric structure information of the occluded area is completed to obtain complete geometric structure information.

[0125] In some alternative implementations, the verification module 203 is further specifically used for: Acquire scene feature information, which includes the identification information of the target object and / or the type information of the current task; Based on scene feature information, determine the limb movement parameters corresponding to the scene feature information; Based on limb movement parameters, the rationality of limb movement information is verified, and the rationality verification results are obtained.

[0126] In some alternative implementations, the verification module 203 is further specifically used for: Acquire motion performance data of the target object and information on the current execution stage of the task; Determine the initial limb movement parameters based on scene feature information; Based on motion performance data, determine the physiological state of the target object; Obtain a pre-established mapping table, which includes the mapping relationship between physiological state, execution stage information and limb movement parameter adjustment amount; Based on physiological state, execution stage information, initial limb movement parameters, and mapping relationship table, the adjustment amount of the initial limb movement parameters is calculated. Based on the adjustment amount, the initial limb movement parameters are adjusted to obtain the limb movement parameters.

[0127] In some alternative implementations, the verification module 203 is further specifically used for: Define conditional rules, which are used to map physiological state and execution stage information into specific adjustment amounts for limb movement parameters. Conditional rules define the parameter adjustment logic under different input combinations. Based on the conditional rules, physiological state and execution stage information are mapped to the adjustment amount of limb movement parameters, thereby establishing a mapping relationship between physiological state, execution stage information and limb movement parameter adjustment amount.

[0128] The limb movement monitoring system based on visual information provided in this embodiment of the invention can execute the limb movement monitoring method based on visual information provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the various modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0129] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0130] The following is a detailed reference. Figure 3 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 302 or a program loaded from memory 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device. The processor 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0131] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0132] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a memory 308, or installed from a ROM 302. When the computer program is executed by the processor 301, it performs the functions defined in the visual information-based limb movement monitoring method of the embodiments of the present invention.

[0133] Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0134] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the visual information-based limb movement monitoring method shown in the above embodiments is implemented.

[0135] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0136] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method of monitoring limb movement based on visual information, characterized by, The method comprises: acquiring visual information of multiple perspectives of a target object; inferring limb movement information of the target object based on the visual information; verifying rationality of the limb movement information based on a preset human body movement rule database to obtain a rationality verification result of current limb movement of the target object; determining whether to trigger a danger judgment instruction based on the rationality verification result.

2. The method of claim 1, wherein, The inference of the limb movement information of the target object based on the visual information comprises: acquiring first three-dimensional space information of the target object in a current working environment; acquiring second three-dimensional space information of an ambient light source of the current working environment; acquiring sensor information of an image sensor used to collect the visual information; predicting an area affected by light reflection in an image of each perspective based on the first three-dimensional space information, the second three-dimensional space information, and the sensor information; generating a visual information reliability indication based on the predicted area; weight adjusting the visual information based on the visual information reliability indication; inferring the limb movement information of the target object based on the weight-adjusted visual information.

3. The method of claim 2, wherein, The prediction of the area affected by light reflection in the image of each perspective based on the first three-dimensional space information, the second three-dimensional space information, and the sensor information comprises: determining geometric structure information of a surface of the target object; determining local light reflection characteristics of the surface of the target object; determining a reflection path of light on the surface of the target object based on the geometric structure information, the local light reflection characteristics, the second three-dimensional space information, and the sensor information; determining the area affected by light reflection in the image based on a propagation direction and intensity of the reflected light.

4. The method of claim 3, wherein, The determination of the local light reflection characteristics of the surface of the target object comprises: acquiring illumination information of the target object under different lighting conditions; acquiring observation information of the target object under different observation angles; determining a reflection performance of the surface of the target object to light based on the illumination information and the observation information; determining the local light reflection characteristics of the surface of the target object based on the reflection performance.

5. The method of claim 3, wherein, When part of the surface of the target object is shielded when the geometric structure information is acquired, the determination of the geometric structure information of the surface of the target object comprises: identifying a position of a shielding subject; determining a shielded area of the target object based on the position of the shielding subject and part of the acquired geometric structure information of the target object; acquiring historical geometric structure information of the target object at a previous time; acquiring reference visual information from other perspectives; completing the geometric structure information of the shielded area based on the reference visual information and the historical geometric structure information to obtain complete geometric structure information.

6. The method of claim 1, wherein, The verification of the rationality of the limb movement information based on the preset human body movement rule database to obtain the rationality verification result of the current limb movement of the target object comprises: acquiring scene feature information, the scene feature information comprising identification information of the target object and / or type information of a current work task; determine, based on the scene feature information, a limb movement parameter corresponding to the scene feature information; perform rationality verification on the limb movement information based on the limb movement parameter, to obtain a rationality verification result.

7. The method of claim 6, wherein, The determining, based on the scene feature information, of a limb movement parameter corresponding to the scene feature information comprises: obtaining movement performance data of a target object and execution stage information of a current task; determining an initial limb movement parameter based on the scene feature information; judging a physiological state of the target object based on the movement performance data; obtaining a pre-established mapping relationship table, the mapping relationship table comprising a mapping relationship between a physiological state, execution stage information and an adjustment amount of a limb movement parameter; calculating an adjustment amount of the initial limb movement parameter based on the physiological state, the execution stage information and the initial limb movement parameter and the mapping relationship table; and adjusting the initial limb movement parameter based on the adjustment amount to obtain the limb movement parameter.

8. The method of claim 7, wherein, The mapping relationship table is established by the following steps: defining a condition rule for mapping the physiological state and the execution stage information to a specific adjustment amount of the limb movement parameter, the condition rule defining parameter adjustment logic under different input combinations; mapping the physiological state and the execution stage information to an adjustment amount of a limb movement parameter according to the condition rule, to establish a mapping relationship between the physiological state, the execution stage information and the adjustment amount of the limb movement parameter.

9. A system for monitoring limb movement based on visual information, characterized in that The system comprises: an acquisition module configured to acquire visual information of multiple perspectives containing a target object; an inference module configured to infer limb movement information of the target object based on the visual information; a verification module configured to perform rationality verification on the limb movement information based on a pre-set human movement rule database, to obtain a rationality verification result of current limb movement of the target object; and a triggering module configured to determine whether to trigger a danger judgment instruction based on the rationality verification result.

10. An electronic device, comprising: comprise: a memory and a processor, which are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method of any one of claims 1 to 8.

Images