AR gesture recognition and non-standard movement warning system for care operations

Abstract

The present application relates to the technical field of motion analysis, in particular to an AR gesture recognition and non-standard action early warning system for nursing operation, a graph Laplacian matrix reflecting spatial topological adjacency relationship is constructed in a video sampling frame, and it is combined with a local operation cumulative load vector determined based on differential optical flow to construct an adjacent sector correlation feature vector, objective quantification of spatial topological conduction and deformation diffusion influence of local wiping operation on adjacent skin area is realized, and the defect that continuous epidermis is forcibly split into independent statistical units in the prior art is overcome; meanwhile, optimal transmission solution is performed on the adjacent sector correlation feature vector and a preset uniform distribution vector, and unified analysis is carried out in combination with physical avoidance constraints to obtain a minimum cost evaluation value, false alarm and missed alarm caused by isolated comparison of local visual features are effectively avoided, and the accuracy of non-standard action early warning of the system through the minimum cost evaluation value is higher.

Description

Technical Field

[0001] This invention relates to the field of motion analysis technology, specifically to an AR gesture recognition and non-standard movement early warning system for nursing procedures. Background Technology

[0002] In nursing care monitoring and AR-assisted teaching scenarios targeting the peripheral area of ​​lesions, operators are typically required to perform a circular wiping motion with uniform force along the outer edge of the lesion. To quantify the standardization of such nursing gestures, existing technologies usually rely on isolated comparisons of local visual features to issue warnings for non-standard actions. Specifically, this involves independently extracting optical flow or pixel displacement data from various local areas around the lesion using visual sensors and calculating the variance of feature data between isolated areas to measure the uniformity of action distribution and issue warnings for non-standard actions based on this uniformity.

[0003] However, this assessment method forcibly divides the continuous annular epidermis into independent statistical units, without considering the spatial topological transmission effect generated by physical wiping on the continuous flexible epidermis. That is, local strong contact will inevitably produce a related deformation effect on the surrounding adjacent skin areas. Furthermore, it fails to combine the physical avoidance constraints that cannot be crossed at the lesion center for overall analysis, and cannot objectively quantify the implicit load of local wiping action spreading to the surrounding non-directly contacted areas. As a result, the accuracy of existing technology in providing early warning of non-standard actions through isolated comparison of local visual features is low. Summary of the Invention

[0004] To address the low accuracy of existing technologies that rely on isolated comparison of local visual features for non-standard movement warnings, this invention aims to provide an AR gesture recognition and non-standard movement warning system for nursing procedures. The specific technical solution adopted is as follows: The first aspect of this invention provides an AR gesture recognition and non-standard movement warning system for nursing operations, comprising: The data acquisition module is used to acquire the fingertip contact associated region and the peripheral annular region of the target lesion in each video sampling frame; and to divide the peripheral annular region of the lesion into at least two local fan-shaped regions. The vector determination module is used to construct a graph Laplacian matrix in each video sampling frame based on the spatial topological adjacency relationship and center distance distribution of each local sector region; and to determine the local operation cumulative load vector based on the differential optical flow of the fingertip contact associated region relative to the annular region of the lesion. The early warning module is used to construct an adjacent sector association feature vector based on the graph Laplacian matrix and the local operation cumulative load vector; perform optimal transmission solution on the adjacent sector association feature vector and the preset uniform distribution vector to determine the minimum cost evaluation value; and issue an early warning for non-standard actions based on the minimum cost evaluation value.

[0005] Furthermore, the process of obtaining the graph Laplacian matrix includes: Each local sector region is sequentially taken as the target region, the local sector regions adjacent to the target region are taken as the adjacent sector regions, and other local sector regions outside the adjacent sector regions are taken as the reference sector regions. Set the edge weight values ​​between the target region and each reference sector region to preset non-adjacent constants; The Euclidean distance between the centroid coordinates of the target region and the centroid coordinates of each adjacent sector region is negatively correlated to determine the corresponding edge weight values. Construct a spatial adjacency matrix; the rows and columns of the spatial adjacency matrix represent local sector regions; the element value of each element position in the spatial adjacency matrix is ​​the edge weight value between the local sector region of the corresponding row and the local sector region of the corresponding column; The corresponding graph Laplace matrix is ​​determined based on the spatial adjacency matrix.

[0006] Furthermore, the process of obtaining the local operation cumulative load vector includes: Obtain the six-DOF pose matrix of each video sampling frame and determine the corresponding inverse pose matrix; perform perspective transformation on each video sampling frame using the inverse pose matrix to align each video sampling frame to the corresponding previous video sampling frame; calculate the local optical flow vector of the fingertip contact associated region and the background optical flow vector of the annular region surrounding the lesion in each aligned video sampling frame; subtract the background optical flow vector from the local optical flow vector to determine the differential optical flow vector of each video sampling frame, and calculate the pixel magnitude of the differential optical flow vector; Obtain the absolute depth value corresponding to the fingertip contact associated area, and calculate the physical scale mapping coefficient in combination with the camera focal length; multiply the physical scale mapping coefficient by the pixel modulus to determine the physical feature modulus of each video sampling frame; The centroid coordinates of the fingertip contact associated regions of all video sampling frames are used as associated coordinates. All video sampling frames are traversed, and the local operation cumulative load value of each local sector is determined based on the cumulative value of the physical feature magnitude of all video sampling frames corresponding to all associated coordinates falling within each local sector. The local operation cumulative load values ​​of all local sector regions are normalized and arranged to construct the local operation cumulative load vector.

[0007] Furthermore, the process of obtaining the associated feature vectors of adjacent sectors includes: The graph Laplacian matrix is ​​subjected to matrix exponent calculation operation in combination with a preset smoothing decay constant to determine the feature transmission matrix; the feature transmission matrix is ​​multiplied with the local operation cumulative load vector to output the adjacent sector association reference vector; the adjacent sector association reference vector is normalized to determine the adjacent sector association feature vector.

[0008] Furthermore, the process of obtaining the minimum cost evaluation value includes: The ideal distribution feature value is determined based on the reciprocal of the total number of local sector regions; a preset uniform distribution vector is obtained, wherein all elements in the preset uniform distribution vector are ideal distribution feature values ​​and the number of corresponding elements is the same as the number of elements in the adjacent sector association feature vector; Construct the shortest ring connectivity path cost matrix based on the Euclidean distance between the centroid coordinates of each local sector region; The adjacent sector association feature vector is used as the source quality distribution, the preset uniform distribution vector is used as the target quality distribution, and the shortest ring connectivity path cost matrix is ​​used as the transfer cost boundary constraint. These are input into the optimal transmission distance algorithm, and the minimum cost evaluation value is output.

[0009] Furthermore, the process of obtaining the shortest ring-connected path cost matrix includes: Connect the centroid coordinates of all adjacent local sector regions in sequence to determine the line segments connecting the centroids of all sectors; Other local sector-shaped regions outside the target region are used as the corresponding transfer regions; In the clockwise direction, the first transfer generation value is determined based on the cumulative length of all sector centroid connecting line segments between the target region and each transfer region; in the counterclockwise direction, the second transfer generation value is determined based on the cumulative length of all sector centroid connecting line segments between the target region and each transfer region; the minimum value between the first transfer generation value and the second transfer generation value is determined based on the corresponding shortest circular connectivity path cost; wherein, the shortest circular connectivity path cost between the target region and itself is 0. Construct a shortest circular connected path cost matrix; the rows and columns of the shortest circular connected path cost matrix represent local sector regions, and the element value of each element position of the shortest circular connected path cost matrix is ​​the shortest circular connected path cost between the local sector region in the corresponding row and the local sector region in the corresponding column.

[0010] Furthermore, the process of issuing early warnings for irregular actions based on the minimum cost evaluation value includes: The minimum cost assessment value is normalized to determine the action warning value; When the action warning value is less than the preset warning threshold, the operation is deemed to be in compliance and no warning signal is triggered. When the action warning value is greater than or equal to the preset warning threshold, the operation is determined to be non-standard, and a non-standard action warning signal is triggered.

[0011] Furthermore, the process of determining the corresponding graph Laplacian matrix based on the spatial adjacency matrix includes: Calculate the degree matrix of the spatial adjacency matrix; determine the graph Laplacian matrix by subtracting the spatial adjacency matrix from the degree matrix.

[0012] Furthermore, the optimal transmission distance algorithm adopts the bulldozer distance algorithm.

[0013] Furthermore, the process of obtaining the local sector region includes: Using the centroid coordinates of the annular region surrounding the lesion as the pole, the annular region surrounding the lesion is divided into at least two local sector regions according to a preset isopolar angle increment.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor. The memory is used to store computer program code, and the processor is used to call and run the computer program code from the memory to execute a system as described in the first aspect or any embodiment of the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer program product comprising computer program code, which, when executed, performs a system as described in the first aspect or any embodiment of the first aspect of the present invention.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing computer program code that, when executed, performs a system as described in the first aspect or any embodiment of the first aspect of the present invention.

[0017] The present invention has the following beneficial effects: This invention constructs a graph Laplacian matrix reflecting spatial topological adjacency relationships in video sampling frames and combines it with a local operation cumulative load vector determined based on differential optical flow to construct an adjacent sector association feature vector. This achieves objective quantification of the spatial topological transmission and deformation diffusion effects of local wiping operations on surrounding adjacent skin areas, overcoming the defect of existing technologies that forcibly divide continuous epidermis into independent statistical units. Simultaneously, the system performs optimal transmission solution on the adjacent sector association feature vector and a preset uniform distribution vector, and performs comprehensive analysis in conjunction with physical avoidance constraints to obtain a minimum cost evaluation value. This effectively avoids false alarms and missed alarms caused by isolated comparison of local visual features, making the system more accurate in issuing warnings of non-standard actions based on the minimum cost evaluation value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a structural diagram of an AR gesture recognition and non-standard movement early warning system for nursing operations, provided in one embodiment of the present invention. Figure 2 This is a schematic diagram of a computer device structure provided in one embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an AR gesture recognition and non-standard movement early warning system for nursing operations proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment, and specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] The following description, in conjunction with the accompanying drawings, details a specific solution for an AR gesture recognition and non-standard movement early warning system for nursing operations provided by the present invention.

[0023] This invention provides an AR gesture recognition and non-standard movement warning system for nursing operations. Please refer to [link / reference]. Figure 1 The diagram illustrates a structural representation of an AR gesture recognition and non-standard movement warning system for nursing procedures, according to an embodiment of the present invention. The system includes: The data acquisition module 101 is used to acquire the fingertip contact associated region and the peripheral annular region of the target lesion in each video sampling frame; and to divide the peripheral annular region of the lesion into at least two local fan-shaped regions. This invention first connects to a medical database via a communication interface to retrieve the patient's historical electronic medical records and image archives. From these, a pre-archived 3D point cloud model of the stoma or pressure ulcer tissue and its 3D boundary contour are extracted and used as the 3D spatial reference for the target lesion. While the operator wears the AR device to perform nursing actions, the system activates the front-mounted RGB-D depth camera to acquire a real-time video stream from the nursing perspective at a constant frame rate, resulting in continuous video sampling frames. To align the 3D lesion data with the current 2D nursing perspective, the system acquires the AR device's six-degree-of-freedom pose matrix and camera intrinsic parameters in real time. Perspective projection transformation is used to map the 3D boundary contour of the target lesion onto the 2D image coordinate system of each video sampling frame. Subsequently, in each video sampling frame, starting from the mapped 2D boundary contour, a preset annular width is offset radially outward from the healthy skin, generating a safe operating zone surrounding the target lesion on the 2D image plane. This zone is precisely defined as the outer annular region of the lesion in the current video sampling frame.

[0024] After completing the two-dimensional mapping and establishment of the annular region surrounding the lesion, the system simultaneously captures the operator's dynamic contact position. For each video sampling frame, the system calls a pre-trained hand skeletal keypoint recognition model to track and lock the pixel coordinates of the fingertip of the operator's index finger performing the wiping action in the two-dimensional image in real time. To ensure the realism of the visual acquisition, the system combines the synchronous depth map output by the depth camera and compares the fingertip depth value at the pixel coordinate with the surrounding skin surface depth value to determine whether physical contact has occurred. When actual contact is confirmed, the system uses the fingertip pixel coordinate as the center to extract a two-dimensional pixel bounding box of a preset size; subsequently, the system iterates through and extracts the depth values ​​of each pixel within the two-dimensional pixel bounding box, extracting and performing connected component analysis on pixels whose absolute difference between the depth value and the aforementioned fingertip depth value is less than a preset contact threshold, thereby accurately extracting the set of real skin pixels corresponding to the largest connected component, and determining this set of real skin pixels as the fingertip contact-related region in the current video sampling frame. Through the above process, the system specifically and clearly obtains the fingertip contact-related region and the annular region surrounding the lesion in each video frame.

[0025] In one specific implementation of this invention, the preset annular width is set to a range of 3.0 to 5.0 cm, preferably 4.0 cm, and can be adjusted according to the specific implementation environment. The preset annular width is used to characterize the physical span limitation of the safety operating zone around the lesion. When the edge of the damaged tissue around the lesion is unclear or the patient's skin is loose, resulting in a wide range of pressure impact, the preset annular width is larger to ensure that all potentially vulnerable stress areas are included in the monitoring range. Conversely, when the edge of the lesion is clear and the surrounding skin is tight and healthy, or when the lesion itself has a very small area of ​​limited area, the preset annular width is smaller to avoid introducing irrelevant distant healthy skin areas that interfere with the assessment accuracy. The preset size is set to 64×64 to 256×256 pixels, with 128×128 pixels being the preferred setting. It can be adjusted according to the specific implementation environment. The preset size is used to characterize the search boundary span of the local contact features of the fingertip. When the operator is wearing heavy nursing gloves, or when performing large-area application that significantly increases the contact area of ​​the fingertip and surrounding area, the preset size is set to be larger to fully cover the set of skin pixels that are deformed by force. Conversely, when the operator is performing fine and precise local wiping with bare hands, or when the depth camera is far from the hand and the proportion of the fingers in the image is small, the preset size is set to be smaller to reduce the redundancy of image connected component clustering calculation caused by the background skin. The preset contact threshold is set to a range of 1.0 mm to 5.0 mm, preferably 2.0 mm, and can be adjusted according to the specific implementation environment. The preset contact threshold is used to characterize the depth tolerance limit for determining effective physical contact between the fingertip and the skin surface. When the inherent environmental noise of the depth camera sensor is large, or when there are large natural physiological undulations and wrinkles on the patient's body surface, the preset contact threshold is set to a larger value to accommodate reasonable surface unevenness and prevent the real pressing and wiping action from being misjudged as a suspended action. Conversely, when using a high-precision depth sensing device, or when it is necessary to strictly monitor weak contact states in flat skin areas, the preset contact threshold is set to a smaller value, thereby accurately eliminating any invalid free-floating actions that do not actually adhere to the skin and improving the rigor of contact determination.

[0026] Furthermore, considering that the periphery of the lesion is a continuous flexible epidermal space, it is difficult to directly extract discrete feature matrices for subsequent mathematical graph theory topological space calculations. Therefore, in order to establish standardized spatial statistical units and realize discrete dimensionality reduction processing of operational features, this embodiment of the invention divides the annular region around the lesion into at least two local sector regions. Specifically, taking the centroid coordinates of the annular region around the lesion as the pole, the annular region around the lesion is cut into at least two local sector regions according to a preset isopolar angle increment. In one specific implementation of this invention, a ray pointing horizontally to the right from the pole to the two-dimensional image plane is taken as the zero-degree starting polar axis. The preset isopolar angle increment is set to a range of 10 to 30 degrees, which can be adjusted according to the specific implementation environment. However, it must be ensured that 360 degrees is divisible by the preset isopolar angle increment, thereby achieving seamless closure and equal division of the annular region around the lesion. In this embodiment, it is set to 30 degrees. The preset isopolar angle increment characterizes the spatial resolution and regional discretization fineness of the evaluation system. When the nursing operation is a fine micro-wiping and the evaluation accuracy of the local stress distribution around the lesion is extremely high, the value of the preset isopolar angle increment is small to increase the number of local fan-shaped regions, thereby more sensitively capturing the operation load fluctuations within a small range. Conversely, when only a basic large-scale coarse uniformity evaluation is required, or when the system's edge computing resources are limited and the real-time data processing frame rate needs to be accelerated, the value of the preset isopolar angle increment is large to reduce the number of divided regions, thereby reducing the overall computational burden of downstream matrix solving.

[0027] The vector determination module 102 is used to construct a graph Laplacian matrix in each video sampling frame based on the spatial topological adjacency relationship and center distance distribution of each local sector region; and to determine the local operation cumulative load vector based on the differential optical flow of the fingertip contact associated region relative to the lesion peripheral annular region.

[0028] Furthermore, considering that isolating the operational load of each local fan-shaped region cannot objectively reflect the real stress transmission effect generated by physical wiping on flexible skin, the hidden damage caused by local strong contact to the surrounding uncontacted areas is easily overlooked by the system. Therefore, in each video sampling frame, the present invention first constructs a graph Laplacian matrix based on the spatial topological adjacency relationship and center distance distribution of each local fan-shaped region. By transforming the discrete physical geometric distribution into a mathematical matrix with topological properties, the subsequent feature calculation can be constrained by the actual physical connectivity of adjacent tissues, thus laying a rigorous spatial constraint foundation for quantifying the smooth diffusion effect of the action distribution on the continuous epidermis.

[0029] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining the graph Laplacian matrix includes: Each local sector is sequentially designated as the target region, the local sector adjacent to the target region is designated as the adjacent sector, and other local sector regions outside the adjacent sector regions are designated as reference sector regions. Since the characteristics of the operational action can only diffuse sequentially along the continuous epidermis to adjacent regions and cannot involve spatial jumps, the local sector regions adjacent to the target region and the non-adjacent local sector regions are processed separately.

[0030] The edge weight values ​​between the target region and each reference sector region are set to preset non-adjacent constants; in this embodiment of the invention, the preset non-adjacent constants are set to 0 to forcibly cut off non-physically continuous feature transmission, ensuring that feature diffusion is strictly limited to the real skin adjacent connectivity relationship.

[0031] The edge weight values ​​are determined by negatively correlated mapping between the centroid coordinates of the target region and the centroid coordinates of each adjacent sector region using the Euclidean distance. In a specific implementation of this invention, the maximum width of the target lesion is calculated across all video sampling frames. The corresponding edge weight values ​​are determined by using the Euclidean distance between the centroid coordinates of the target region and the centroid coordinates of each adjacent sector region as the denominator and the maximum width as the numerator. Since the target region and the adjacent sector regions are physically independent adjacent regions, their centroids do not coincide spatially, so the denominator cannot be 0. By dividing the maximum width by the Euclidean distance, the edge weights retain the negative correlation topological property of greater weight for closer distances while eliminating physical dimensions to meet the mathematical requirements of subsequent matrix exponential operations. This achieves adaptive uniformity for different patient lesion scales.

[0032] A spatial adjacency matrix is ​​constructed; the rows and columns of the spatial adjacency matrix represent local sector regions; the element value of each element in the spatial adjacency matrix is ​​the edge weight between the local sector region in the corresponding row and the local sector region in the corresponding column. In this embodiment of the invention, the local sector regions in the rows and columns of the spatial adjacency matrix are arranged counterclockwise from the polar axis, and this arrangement is used as the default arrangement for subsequent analysis; since the elements on the diagonal of the spatial adjacency matrix correspond to the local sector region and itself, in order to eliminate computational redundancy and topological self-loop interference caused by the repeated diffusion of features to themselves, the values ​​of the elements on the diagonal of the spatial adjacency matrix are all set to 0.

[0033] The corresponding graph Laplacian matrix is ​​determined based on the spatial adjacency matrix. Specifically, the degree matrix of the spatial adjacency matrix is ​​calculated, i.e., a diagonal matrix is ​​constructed where the main diagonal elements are the sum of the values ​​of the corresponding rows in the spatial adjacency matrix, and all non-main diagonal elements are 0. The graph Laplacian matrix is ​​determined by subtracting the spatial adjacency matrix from the degree matrix. Since the degree matrix represents the total divergence weight of the transmission characteristics from each local sector to the surrounding adjacent regions, while the spatial adjacency matrix represents the direct correlation strength between the transmission between regions, the discrete Laplacian operator constructed by subtracting the two can accurately simulate the physical transmission law of local wiping stress from high-load areas to surrounding low-load areas on a continuous flexible skin.

[0034] To further filter out head-shaking noise when the operator wears the AR device and extract pure frictional contact features to objectively reflect the actual cumulative physical displacement applied to the skin surface during nursing operations, this embodiment of the invention determines the local cumulative load vector based on the differential optical flow of the fingertip contact associated area relative to the annular area surrounding the lesion. By separating the motion interference from the global perspective from the local pressure deformation, the extracted displacement features only characterize the actual amount of epidermal slippage caused by the fingertip force, thereby significantly improving the anti-interference ability and accuracy of operation load quantification.

[0035] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining the local operation cumulative load vector includes: The six-degree-of-freedom (DOF) pose matrix of each video sampling frame is obtained, and the corresponding inverse pose matrix is ​​determined. That is, the six-degree-of-freedom pose matrix representing the camera's three-dimensional spatial position and rotational attitude in each video sampling frame is obtained. The six-degree-of-freedom pose matrix of the current video sampling frame is inverted, and the inverted matrix is ​​multiplied by the six-degree-of-freedom pose matrix of the previous video sampling frame to obtain the inverse pose matrix representing the relative motion relationship. The inverse pose matrix is ​​used to perform perspective transformation on each video sampling frame to align each video sampling frame to the corresponding previous video sampling frame. Through the above matrix inversion and multiplication operations, the true relative displacement and rotation transformation process of the camera between two adjacent frames can be accurately calculated. Then, the inverse pose matrix is ​​used to perform reverse perspective mapping to offset the global field of view offset introduced by head shaking and ensure that the coordinate reference for subsequent optical flow calculation is absolutely stationary.

[0036] The local optical flow vector of the fingertip contact associated region and the background optical flow vector of the annular region surrounding the lesion are calculated separately in each aligned video sampling frame. Specifically, in each aligned video sampling frame, the mean of the optical flow vectors of all pixels in the fingertip contact associated region is used as the local optical flow vector; the mean of the optical flow vectors of all pixels in the annular region surrounding the lesion is used as the background optical flow vector.

[0037] The differential optical flow vector for each video sampling frame is determined by subtracting the background optical flow vector from the local optical flow vector, and the pixel magnitude of the differential optical flow vector is calculated. Since the local optical flow vector in the fingertip contact area actually includes the actual frictional sliding displacement of the finger and the overall offset displacement of the surrounding skin due to the traction, while the background optical flow vector in the annular area around the lesion only represents this overall offset displacement, by performing a vector subtraction operation between the two, the overall offset component of the skin can be directly canceled out in the geometric dimension, thereby accurately separating the pure fingertip relative frictional displacement feature.

[0038] The absolute depth value corresponding to the fingertip contact area is obtained, and a physical scale mapping coefficient is calculated by combining it with the camera focal length. This coefficient is the ratio of the absolute depth value to the camera focal length. The physical scale mapping coefficient is then multiplied by the pixel modulus to determine the physical feature modulus of each video sampling frame. This ratio calculation establishes the mapping ratio between the 2D pixel and the real 3D space at the current depth. Furthermore, the multiplication operation eliminates the perspective projection effect, accurately restoring the 2D pixel offset, which lacks practical physical meaning, to a 3D true wiping length with actual physical distance dimensions.

[0039] The centroid coordinates of the fingertip contact associated regions of all video sampling frames are used as associated coordinates. All video sampling frames are traversed, and the local operation cumulative load value of each local sector is determined based on the cumulative value of the physical feature magnitude of all video sampling frames corresponding to all associated coordinates falling within each local sector. The local operation cumulative load values ​​of all local sector regions are normalized and arranged to construct the local operation cumulative load vector.

[0040] This invention calculates the sum of the local operation cumulative load values ​​of all local sector regions to determine the overall cumulative load value. Then, based on the ratio of the local operation cumulative load value of each local sector region to the overall cumulative load value, the corresponding load normalization value is determined. The load normalization values ​​of all local sector regions are arranged in a default order to construct a local operation cumulative load vector. It should be noted that if the overall cumulative load value is 0, it indicates that no effective wiping friction has occurred, and therefore no irregular actions will occur. In this case, subsequent calculations and analyses are stopped directly, and the minimum cost evaluation value is set to 0 to issue an irregular action warning.

[0041] Because the longer the wiping action stays in a certain area, the greater the cumulative stress on the epidermis, the summation of the cumulative load value of this local operation can more objectively reflect the total physical load borne by the local skin. By summing and normalizing the cumulative load value of local operations, the influence of absolute operation time and overall force can be eliminated, and the spatial distribution pattern that purely represents the operation proportion of each sector area can be extracted. Furthermore, this summation and normalization operation makes the sum of the elements in the cumulative load vector of local operations constant to 1, thus strictly satisfying the mathematical conservation constraint in the subsequent optimal transmission distance algorithm that the total source quality distribution and the total target quality distribution must be equal.

[0042] The early warning module 103 is used to construct the adjacent sector association feature vector based on the graph Laplacian matrix and the local operation cumulative load vector; perform optimal transmission solution on the adjacent sector association feature vector and the preset uniform distribution vector to determine the minimum cost evaluation value; and conduct irregular action early warning based on the minimum cost evaluation value.

[0043] Furthermore, considering that the local operation cumulative load vector only objectively quantifies the friction in the area directly in contact with the fingertip, while ignoring the physical traction effect on adjacent tissues after continuous epidermal pressure, this embodiment of the invention constructs an adjacent sector association feature vector based on the graph Laplacian matrix and the local operation cumulative load vector. By introducing a graph theory feature diffusion mechanism, the discrete operation features are smoothly extended along the spatial topological adjacency network, so that the hidden areas on the surface that are not directly wiped but are subjected to high transmitted stress due to strong operations in the surrounding area can be objectively revealed, thereby greatly improving the comprehensiveness and accuracy of the overall operation load spatial distribution assessment.

[0044] Preferably, in some possible implementations of the present invention, the process of obtaining the associated feature vector of adjacent sectors includes: performing a matrix exponentiation operation on the graph Laplace matrix in combination with a preset smoothing decay constant to determine the feature transmission matrix; specifically: multiplying each element in the graph Laplace matrix by the negative of the preset smoothing decay constant to determine the scaled graph Laplace matrix, and then performing a matrix exponentiation function operation on the scaled graph Laplace matrix to determine the feature transmission matrix.

[0045] The preset smoothing attenuation constant here is used to characterize the energy dissipation law and conduction attenuation rate of human epidermal deformation during physical extension. The larger the value, the weaker the ability of the feature to diffuse to the surrounding adjacent sectors. This value can be obtained by performing a single-point compression test on clinical biomimetic skin in advance and fitting its displacement distance attenuation curve. In this embodiment of the invention, the preferred range of the preset smoothing attenuation constant is 0.1 to 0.5, which can be adjusted according to the specific implementation environment. In this embodiment of the invention, it is set to 0.25. When the skin around the patient's lesion is relatively loose and the subcutaneous soft tissue is relatively thick, the deformation caused by physical wiping is easily absorbed locally and difficult to conduct to the periphery. The value of the preset smoothing attenuation constant is larger to limit the excessive diffusion of the feature in the topological space and truly reflect the physical property that the stress is limited to the vicinity of the contact point. Conversely, when the skin around the patient's lesion is tight, thin, or located in the area of ​​bone protrusion, the local surface friction is very likely to cause a large area of ​​traction deformation along the tight epidermis. The value of the preset smoothing attenuation constant is smaller to enhance the spatial diffusion ability during matrix solution, thereby accurately capturing the implicit traction effect borne by a large area of ​​adjacent tissues.

[0046] Multiplying the graph Laplacian matrix by a negative number of a preset smoothing decay constant aims to transform the distance matrix into a decay rate matrix that conforms to the thermodynamic physical dissipation law. This ensures that the transmission of features in the topological network exhibits an exponential decrease with increasing spatial distance, rather than a reverse amplification. Furthermore, the reason for performing matrix exponential function operations on this scaled matrix is ​​that the graph Laplacian matrix only describes the microscopic connectivity of a one-step jump between adjacent nodes in the network, while the matrix exponential operator (which involves high-order power operations on the matrix through Taylor series expansion) can mathematically realize the macroscopic continuous diffusion solution of multiple jumps between any two non-directly adjacent nodes in the network. This rigorously elevates the discrete local connectivity to a final transmission framework that characterizes the smooth penetration of global features.

[0047] It should be noted that the matrix exponent calculation operation in the embodiments of the present invention refers to the matrix exponent function operation based on Taylor series expansion. Its purpose is to achieve accurate solution of the operation characteristics in the multi-step continuous jump and global physical dissipation in the topological network. It belongs to the standard linear algebra operator operation in the field of graph theory and spectral analysis that characterizes the diffusion process of continuous time characteristics. It is not to independently exponentiate each element inside the graph Laplacian matrix. Furthermore, this matrix exponent calculation operation is a technical means well known to those skilled in the art, and will not be further limited or described here.

[0048] The process involves multiplying the feature transfer matrix with the local operation cumulative load vector to output an adjacent sector association reference vector. This adjacent sector association reference vector is then normalized to determine the adjacent sector association feature vector. The normalization method employs sum-based normalization, where the ratio of each element in the adjacent sector association reference vector to the sum of all elements is used as the normalized value for that element. All elements in the adjacent sector association reference vector are then replaced with their corresponding normalized values ​​to determine the adjacent sector association feature vector.

[0049] Through this multiplication operation, the system can use the local operation accumulated load vector as the initial heat source and simulate the natural physical dissipation of load characteristics along the topological connectivity path defined by the graph Laplace matrix. On this basis, by summing all elements in the adjacent sector association reference vector and dividing each element in the reference vector by the summation result to perform sum normalization, not only is the overall numerical expansion that may be caused by matrix diffusion operation effectively filtered out, but the sum of all elements in the output adjacent sector association feature vector is also strictly kept to be 1. Thus, while quantifying the transmission effect, it perfectly satisfies the total conservation constraint of the source quality distribution in the subsequent optimal transmission distance algorithm.

[0050] Furthermore, considering that conventional variance or absolute value difference comparisons can only measure the dispersion of the operational load in numerical distribution, but cannot reflect the actual physical space distance required to transfer redundant wiping actions from high-load areas to low-load areas, and are prone to causing theoretical evaluation paths to cross the lesion center that cannot withstand external forces, this embodiment of the invention performs optimal transmission solution on the adjacent sector correlation feature vector and the preset uniform distribution vector to determine the minimum cost evaluation value; by introducing an optimal transmission distance algorithm that considers physical connectivity costs, the calculated evaluation value can objectively quantify the minimum total transfer space cost required to smoothly schedule the current non-uniform operational distribution state to an ideal uniform state, thereby providing a scientific benchmark that takes into account both topological physical safety and spatial displacement costs for accurately determining the degree of deviation of the wiping action from the norm.

[0051] Preferably, in some possible implementations of the embodiments of the present invention, the process of obtaining the minimum cost evaluation value includes: The ideal distribution characteristic value is determined based on the reciprocal of the total number of local sector regions. A preset uniform distribution vector is obtained, where all elements are ideal distribution characteristic values ​​and the number of corresponding elements is the same as the number of elements in the adjacent sector correlation characteristic vector. Since the sum of the values ​​of the adjacent sector correlation characteristic vectors representing the source quality distribution has been normalized and forcibly aligned to 1, in order to ensure that the sum of the values ​​of the preset uniform distribution vector representing the target quality distribution is strictly conserved, the total distribution quality to be allocated must be 1. The reciprocal value obtained by evenly distributing this total quality to all local sector regions represents the ideal physical equilibrium state where the operating load of each region is absolutely equal. The preset uniform distribution vector constructed in this way provides a theoretically absolute normalized distribution benchmark for subsequent optimal transmission solutions.

[0052] Construct the shortest circular connectivity path cost matrix based on the Euclidean distance between the centroid coordinates of each local sector region. Specifically: connect the centroid coordinates of all adjacent local sector regions in sequence to determine the centroid connection line segments of all sectors; and take other local sector regions outside the target region as the corresponding transfer regions.

[0053] In the clockwise direction, the first transfer cost is determined based on the cumulative length of all sector centroid connecting segments between the target region and each transfer region. In the counterclockwise direction, the second transfer cost is determined based on the cumulative length of all sector centroid connecting segments between the target region and each transfer region. The minimum value between the first and second transfer costs is determined based on the shortest circular path cost. The shortest circular path cost between the target region and itself is 0. By forcing feature transfer to proceed only along the clockwise and counterclockwise circular outer perimeter paths formed by adjacent sector centroid connecting segments, any theoretical straight shortcuts that could directly cross the lesion center are strictly cut off, thus completely eliminating the medical safety risks of scheduling paths crossing vulnerable lesion regions at the algorithm's underlying level. At the same time, taking the minimum value of the clockwise and counterclockwise bidirectional paths objectively simulates the real physical laws of seeking the feature transfer path with the least resistance and shortest distance on a closed annular skin band.

[0054] A shortest-circular connectivity path cost matrix is ​​constructed. The rows and columns of this matrix represent local sector regions, and the element value at each position represents the shortest-circular connectivity path cost between the local sector region in the corresponding row and column. This matrix fully quantifies the precise spatial distance cost required to transfer a feature from one local sector region to another without penetrating physical restricted areas, forming an insurmountable underlying cost constraint in subsequent optimization solutions. In this embodiment, the arrangement order of the local sector regions in the rows and columns of the spatial adjacency matrix is ​​the default order, ensuring that the elements in the generated spatial adjacency matrix strictly map to the circular topological connectivity rules in real 3D space. This not only guarantees that the matrix has a consistent and traceable physical spatial meaning in its mathematical expression but also provides a coordinate index alignment benchmark for correctly calculating the shortest-circular connectivity cost between adjacent regions and the local feature graph theory diffusion.

[0055] The algorithm uses the associated feature vectors of adjacent sectors as the source quality distribution, a preset uniform distribution vector as the target quality distribution, and the shortest circular connectivity path cost matrix as the boundary constraint for transfer costs. These are input into the optimal transmission distance algorithm, which outputs a minimized cost evaluation value. In this embodiment, the optimal transmission distance algorithm employs the bulldozer distance algorithm. This algorithm uses linear programming to find a scheduling scheme that minimizes the total transfer workload (i.e., the sum of the product of the transfer feature quotas of each region and the corresponding shortest circular connectivity path cost). Since the total amounts of the input source quality distribution and target quality distribution have been pre-normalized and aligned to an absolutely conserved state, the bulldozer distance algorithm can converge stably and accurately calculate the minimum global displacement cost required to smoothly fill the currently unevenly stressed associated features to an ideal uniform state. The output minimized cost evaluation value objectively reflects the degree of distortion of the current spatial load. The larger this value, the greater the deviation of the current wiping action distribution from the standard requirements, and the higher the risk of stress imbalance in the tissues surrounding the lesion. It should be noted that the bulldozer distance algorithm is a well-known technique in the art, and will not be further limited or elaborated upon here.

[0056] Finally, an alert for non-standard actions is issued based on the minimum cost evaluation value. Preferably, in some possible implementations of this invention, the process of issuing an alert for non-standard actions based on the minimum cost evaluation value includes: The minimum cost assessment value is normalized to determine the action warning value. In this embodiment of the invention, the method for normalizing the minimum cost assessment value is maximum value normalization. The maximum value in maximum value normalization is determined by analyzing the maximum value of the minimum cost assessment value calculated from lesion samples with the same lesion type as the target lesion in this embodiment of the medical database.

[0057] When the action warning value is less than the preset warning threshold, the operation is deemed to be standard and no warning signal is triggered; when the action warning value is greater than or equal to the preset warning threshold, the operation is deemed to be non-standard and a non-standard action warning signal is triggered. In this embodiment of the invention, the preferred range of the preset warning threshold is set to 0.5 to 0.8, which can be adjusted according to the specific implementation environment. In this embodiment of the invention, it is set to 0.65. The preset warning threshold is used to characterize the system's safe tolerance limit for epidermal deformation damage caused by uneven spatial distribution of nursing wiping actions. When the patient's lesion surrounding skin tissue is healthy, in good recovery condition, and has strong tensile elasticity, or when the current operation belongs to basic cleaning care with relatively broad requirements for uniformity of action, the preset warning threshold is set to a larger value to give the system a higher margin of error; conversely, when the patient's lesion is in the acute postoperative period, the surrounding granulation tissue is extremely fragile and prone to mechanical secondary soft tissue tearing, or when the system is applied to high-precision standardized clinical teaching and assessment scenarios, the preset warning threshold is set to a smaller value to significantly improve the system's monitoring sensitivity.

[0058] In summary, an AR gesture recognition and non-standard action early warning system for nursing operations constructs a graph Laplacian matrix reflecting spatial topological adjacency relationships in video sampling frames and combines it with a local operation cumulative load vector determined based on differential optical flow to construct an adjacent sector association feature vector. This achieves objective quantification of the spatial topological transmission and deformation diffusion effects of local wiping operations on surrounding adjacent skin areas, overcoming the shortcomings of existing technologies that forcibly divide continuous epidermis into independent statistical units. Simultaneously, the system performs optimal transmission solution on this adjacent sector association feature vector and a preset uniformly distributed vector, and conducts comprehensive analysis in conjunction with physical avoidance constraints to obtain a minimum cost evaluation value. This effectively avoids false alarms and missed alarms caused by isolated comparison of local visual features, making the system more accurate in providing non-standard action early warnings through the minimum cost evaluation value.

[0059] This invention also provides a computer device; please refer to [link / reference]. Figure 2 The diagram illustrates a computer device structure according to an embodiment of the present invention. The computer device includes a memory 201, a processor 202, and a computer program 203 stored in the memory 201 and running on the processor 202. When the processor 202 executes the computer program 203, the computer device can execute any of the aforementioned AR gesture recognition and non-standard action warning systems for nursing operations.

[0060] This invention also provides a computer program product that, when run on a computer device, enables the computer device to execute any of the aforementioned AR gesture recognition and non-standard action warning systems for nursing operations.

[0061] This invention also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer device, the computer device can execute any of the aforementioned AR gesture recognition and non-standard action warning systems for nursing operations.

[0062] In the embodiments provided by the present invention, it should be understood that the computer device, computer program product and computer-readable storage medium provided are all used to execute the corresponding system provided above, and therefore the beneficial effects they can achieve can be referred to the beneficial effects of the system provided above, which will not be repeated here.

[0063] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. An AR gesture recognition and non-standard movement early warning system for nursing operations, characterized in that, The system includes: The data acquisition module is used to acquire the fingertip contact associated region and the peripheral annular region of the target lesion in each video sampling frame; and to divide the peripheral annular region of the lesion into at least two local fan-shaped regions. The vector determination module is used to construct a graph Laplacian matrix in each video sampling frame based on the spatial topological adjacency relationship and center distance distribution of each local sector region; and to determine the local operation cumulative load vector based on the differential optical flow of the fingertip contact associated region relative to the annular region of the lesion. The early warning module is used to construct an adjacent sector association feature vector based on the graph Laplacian matrix and the local operation cumulative load vector; perform optimal transmission solution on the adjacent sector association feature vector and the preset uniform distribution vector to determine the minimum cost evaluation value; and issue an early warning for non-standard actions based on the minimum cost evaluation value.

2. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 1, characterized in that, The process of obtaining the graph Laplacian matrix includes: Each local sector region is sequentially taken as the target region, the local sector regions adjacent to the target region are taken as the adjacent sector regions, and other local sector regions outside the adjacent sector regions are taken as the reference sector regions. Set the edge weight values ​​between the target region and each reference sector region to preset non-adjacent constants; The Euclidean distance between the centroid coordinates of the target region and the centroid coordinates of each adjacent sector region is negatively correlated to determine the corresponding edge weight values. Construct a spatial adjacency matrix; the rows and columns of the spatial adjacency matrix represent local sector regions; the element value of each element position in the spatial adjacency matrix is ​​the edge weight value between the local sector region of the corresponding row and the local sector region of the corresponding column; The corresponding graph Laplace matrix is ​​determined based on the spatial adjacency matrix.

3. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 1, characterized in that, The process of obtaining the local operation cumulative load vector includes: Obtain the six-DOF pose matrix of each video sampling frame and determine the corresponding inverse pose matrix; perform perspective transformation on each video sampling frame using the inverse pose matrix to align each video sampling frame to the corresponding previous video sampling frame; calculate the local optical flow vector of the fingertip contact associated region and the background optical flow vector of the annular region surrounding the lesion in each aligned video sampling frame; subtract the background optical flow vector from the local optical flow vector to determine the differential optical flow vector of each video sampling frame, and calculate the pixel magnitude of the differential optical flow vector; Obtain the absolute depth value corresponding to the fingertip contact associated area, and calculate the physical scale mapping coefficient in combination with the camera focal length; multiply the physical scale mapping coefficient by the pixel modulus to determine the physical feature modulus of each video sampling frame; The centroid coordinates of the fingertip contact associated regions of all video sampling frames are used as associated coordinates. All video sampling frames are traversed, and the local operation cumulative load value of each local sector is determined based on the cumulative value of the physical feature magnitude of all video sampling frames corresponding to all associated coordinates falling within each local sector. The local operation cumulative load values ​​of all local sector regions are normalized and arranged to construct the local operation cumulative load vector.

4. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 1, characterized in that, The process of obtaining the associated feature vectors of adjacent sectors includes: The graph Laplacian matrix is ​​subjected to matrix exponent calculation operation in combination with a preset smoothing decay constant to determine the feature transmission matrix; the feature transmission matrix is ​​multiplied with the local operation cumulative load vector to output the adjacent sector association reference vector; the adjacent sector association reference vector is normalized to determine the adjacent sector association feature vector.

5. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 2, characterized in that, The process of obtaining the minimum cost assessment value includes: The ideal distribution feature value is determined based on the reciprocal of the total number of local sector regions; a preset uniform distribution vector is obtained, wherein all elements in the preset uniform distribution vector are ideal distribution feature values ​​and the number of corresponding elements is the same as the number of elements in the adjacent sector association feature vector; Construct the shortest ring connectivity path cost matrix based on the Euclidean distance between the centroid coordinates of each local sector region; The adjacent sector association feature vector is used as the source quality distribution, the preset uniform distribution vector is used as the target quality distribution, and the shortest ring connectivity path cost matrix is ​​used as the transfer cost boundary constraint. These are input into the optimal transmission distance algorithm, and the minimum cost evaluation value is output.

6. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 5, characterized in that, The process of obtaining the cost matrix of the shortest ring-connected path includes: Connect the centroid coordinates of all adjacent local sector regions in sequence to determine the line segments connecting the centroids of all sectors; Other local sector-shaped regions outside the target region are used as the corresponding transfer regions; In the clockwise direction, the first transfer generation value is determined based on the cumulative length of all sector centroid connecting line segments between the target region and each transfer region; in the counterclockwise direction, the second transfer generation value is determined based on the cumulative length of all sector centroid connecting line segments between the target region and each transfer region; the minimum value between the first transfer generation value and the second transfer generation value is determined based on the corresponding shortest circular connectivity path cost; wherein, the shortest circular connectivity path cost between the target region and itself is 0. Construct a shortest circular connected path cost matrix; the rows and columns of the shortest circular connected path cost matrix represent local sector regions, and the element value of each element position of the shortest circular connected path cost matrix is ​​the shortest circular connected path cost between the local sector region in the corresponding row and the local sector region in the corresponding column.

7. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 1, characterized in that, The process of issuing early warnings for irregular actions based on the minimum cost assessment value includes: The minimum cost assessment value is normalized to determine the action warning value; When the action warning value is less than the preset warning threshold, the operation is deemed to be in compliance and no warning signal is triggered. When the action warning value is greater than or equal to the preset warning threshold, the operation is determined to be non-standard, and a non-standard action warning signal is triggered.

8. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 2, characterized in that, The process of determining the corresponding graph Laplacian matrix based on the spatial adjacency matrix includes: Calculate the degree matrix of the spatial adjacency matrix; determine the graph Laplacian matrix by subtracting the spatial adjacency matrix from the degree matrix.

9. The AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 5, characterized in that, The optimal transmission distance algorithm adopts the bulldozer distance algorithm.

10. An AR gesture recognition and non-standard movement early warning system for nursing operations according to claim 1, characterized in that, The process of obtaining the local sector-shaped region includes: Using the centroid coordinates of the annular region surrounding the lesion as the pole, the annular region surrounding the lesion is divided into at least two local sector regions according to a preset isopolar angle increment.

Images