A production line personnel action recognition method based on a video understanding algorithm

Abstract

The application relates to the technical field of action recognition, in particular to a production line personnel action recognition method based on a video understanding algorithm, which comprises the following steps: capturing continuous actions of production line personnel by using a camera, and performing feature extraction on the captured continuous actions; constructing an action recognition model based on a video understanding algorithm, and performing action recognition on the extracted motion features; and performing optimization training on the action recognition result based on a feature fusion algorithm; the application can capture the actions of personnel on a production line in real time through a camera, analyze and recognize the actions, timely issue a warning when non-standard actions are recognized, and reduce the possibility of equipment damage; the application can realize accurate recognition of complex actions by combining a convolutional neural network and a long short-term memory network to construct an action recognition model, effectively improve the recognition accuracy, and improve the applicability and generalization ability of the model on different production lines through optimization training on the recognition result by using the feature fusion algorithm.

Description

Technical Field

[0001] This invention relates to the technical field of action recognition, specifically to a method for recognizing the actions of production line personnel based on video understanding algorithms. Background Technology

[0002] Video-based human motion recognition methods can monitor workers' operational behavior on production lines in real time, helping companies to promptly identify and correct non-standard operations, reduce human error, and avoid problems such as production line downtime or equipment damage. Simultaneously, identifying and issuing warnings for dangerous actions can effectively prevent safety accidents and ensure worker safety. However, current human motion recognition methods face the following challenges:

[0003] Existing video understanding algorithms have a high recognition rate when dealing with simple static or single actions, but the accuracy of the algorithms may drop significantly when the actions are more complex, fast, or continuous.

[0004] Action recognition models typically require a large amount of labeled data for training. However, the actions of personnel on different production lines may have certain specificities and differences. Existing models have poor generalization ability, and when applied to new production environments, the recognition accuracy may decrease, making it difficult to apply them universally across different production line scenarios.

[0005] Although existing technologies can use video data for motion recognition, single visual information often cannot fully reflect the operational details of production line personnel, such as the force of operation and the use of tools, resulting in limitations in recognition effectiveness. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for recognizing the actions of production line personnel based on video understanding algorithms, comprising the following steps:

[0008] The system uses cameras to capture the continuous movements of production line workers and extracts features from the captured movements.

[0009] An action recognition model is constructed based on video processing algorithms to perform action recognition on the extracted motion features, specifically as follows:

[0010] The convolutional neural network algorithm is used to perform convolution operations on the frame sequence of the video, and the long short-term memory network is used to capture the temporal dependence of the extracted action features. Finally, the two calculation results are combined to construct an action recognition model.

[0011] The action recognition results are optimized and trained based on a feature fusion algorithm, specifically as follows:

[0012] Based on the action recognition results, the probability of each type of action is calculated, and the loss function is used to optimize the training based on the calculated probability results.

[0013] As a preferred embodiment of the production line personnel action recognition method based on video understanding algorithm described in this invention, the feature extraction of the captured continuous actions is specifically as follows:

[0014] Based on the frame information from the video decomposition, the human motion features are extracted, as follows:

[0015] Based on the property that the pixel intensity of an object remains constant over time during motion, constraining the pixel intensity in the frame information yields the following:

[0016] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)

[0017] Where I(x,y,t) represents the pixel brightness at position (x,y) at time t, Δx and Δy represent the spatial displacement of the pixel, and Δt represents the time interval;

[0018] If we perform displacement decomposition on the information of adjacent frames, then we have:

[0019]

[0020] Where u and v represent the velocity components of the pixel in the x and y directions, respectively. These represent the gradients of the image in the spatial dimension. This represents the gradient of the image over time.

[0021] As a preferred embodiment of the production line personnel action recognition method based on video understanding algorithm described in this invention, wherein: by constructing an optimized objective function to extract the motion information of each pixel, then,

[0022]

[0023] Where u and v represent the velocity components of the pixel in the x and y directions, respectively. These represent the gradients of the image in the spatial dimension. This represents the gradient of the image over time. The ratios represent the smoothness of the pixel's velocity change in the x and y directions, respectively, and λ represents the smoothing term coefficient, which is used to adjust the balance between the data terms and the smoothing term.

[0024] By iteratively optimizing the objective function until all video frames at all times have been iterated, the velocity information u(x,y) and v(x,y) of each pixel in the video can be obtained.

[0025] As a preferred embodiment of the production line worker motion recognition method based on video understanding algorithm described in this invention, wherein: based on the acquired velocity information u(x,y) and v(x,y) of each pixel, the amplitude and direction of the motion are calculated, then...

[0026] Calculation of motion amplitude

[0027]

[0028] Calculation of direction of motion

[0029]

[0030] Where ||V(x,y)|| represents the motion amplitude of the pixel, is the magnitude of the motion speed, and θ(x,y) represents the calculated motion direction;

[0031] Based on the calculated speed and direction of motion, each frame of video data (I) can be extracted. n The motion characteristics in the data are used to form a time series {V1,V2,...,V...} T}, where V T This represents the motion characteristics of the T-th frame.

[0032] As a preferred embodiment of the production line personnel action recognition method based on video understanding algorithm described in this invention, the specific steps of performing convolution operation on the video frame sequence using a convolutional neural network algorithm are as follows:

[0033] The input video frame sequence M = {I1, I2, ..., I...} n}, where I n This represents the nth frame of the video, where n represents the total number of frames in the current video.

[0034] By using convolutional neural networks to perform convolution operations in both spatial and temporal dimensions, we have:

[0035]

[0036] Among them, h t+1 (x,y) represents the convolution output value at time t+1, σ represents the activation function, which is set by the implementer according to the actual application scenario, w(i,j,t) represents the weights of the neural network convolution kernel, and i, j, and t represent the spatial and temporal indices of the convolution kernel parameters, respectively. t(x+i,y+j) represents the pixel value at coordinates (x+i,y+j) in the t-th frame of the input video frame sequence M. b represents the pixel intensity of the original video frame after spatial and temporal displacement. k represents the time step size, which is the size of the convolution kernel in the time dimension. r represents the spatial step size, which is the size of the convolution kernel in the spatial dimension.

[0037] As a preferred embodiment of the production line personnel action recognition method based on video understanding algorithm described in this invention, the extraction of action features using a Long Short-Term Memory (LSTM) network for time-dependent capture is as follows:

[0038] The extracted action feature sequence {V1,V2,...,V T As input to the Long Short-Term Memory network, x is then obtained. t =V t , where x t This represents the amount of input to the Long Short-Term Memory (LSTM) network;

[0039] Based on input quantity x t and the output h of the convolutional neural network t-1 Given (x, y), calculate the output of the forget gate, then we have:

[0040] f t =σ(W f ·[h t-1 (x,y),x t ]+b f )

[0041] Among them, f t W represents the output of the forget gate at the current moment. f h represents the weight matrix of the forget gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. f σ represents the bias term of the forget gate, and σ represents the activation function, which is set by the implementer according to the actual application scenario.

[0042] By updating memory units using input gates and candidate memories, we have:

[0043] i t =σ(W i ·[h t-1 (x,y),x t ]+b i )

[0044]

[0045] Among them, i tW represents the output result of the input gate. i h represents the weight matrix of the input gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. i σ represents the bias term of the input gate, and σ represents the activation function, which is set by the implementer according to the actual application scenario. W represents the candidate memory value at the current moment. C Let b represent the weight matrix of the candidate memory, tanh represent the hyperbolic tangent function, and b represent the weight matrix of the candidate memory. C Bias terms representing candidate memories;

[0046] Updating the memory unit results in:

[0047]

[0048] Among them, C t f represents the memory unit at the current moment. t C represents the output of the forget gate at the current moment. t-1 i represents the memory unit from the previous time step. t This indicates the input gate's output result. This represents the candidate memory value at the current moment;

[0049] Based on the updated memory units, the output result is calculated, and then we have:

[0050] O t =tanh(C t )*σ(W O ·[h t-1 (x,y),x t ]+b O )

[0051] Among them, O t W represents the output result of the calculation. O h represents the weight matrix of the output gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. O This represents the output bias term, and σ represents the activation function, which is set by the implementer according to the actual application scenario.

[0052] Based on the calculated output, the actions are classified, and then...

[0053] y = Softmax(W o ·h t (x,y)+b O )

[0054] Among them, W O h represents the weight matrix of the output gate. t (x,y) represents the output of the neural network at the current time step, b O This represents the output bias term, where y represents the classification and recognition result corresponding to the action feature.

[0055] As a preferred embodiment of the production line personnel action recognition method based on video understanding algorithm described in this invention, the calculation of the probability of various actions is specifically as follows:

[0056] Based on the action recognition results, the probabilities of various actions are calculated, and then we have:

[0057]

[0058] Among them, W c b represents the weight coefficient for action category c in action recognition. c Let y denote the bias term, C denote the total number of categories in action recognition, x denote the input action feature, and P(y = c|x) denote the probability of category c in action recognition.

[0059] As a preferred embodiment of the production line personnel action recognition method based on video understanding algorithm described in this invention, the optimization training using a loss function is specifically as follows:

[0060]

[0061] Where P(y=i|x) represents the probability of category i in action recognition, i represents the index of action recognition type, C represents the total number of categories in action recognition, L represents the output of the loss function, and L is the difference between the output of the recognition model and the true label;

[0062] Optimizing the loss function involves repeatedly calculating its output value until it converges to a stable low value. Then, we have...

[0063] When the output of the loss function satisfies |L t+1 -L t When |→0, it means that the current loss function has reached maximum convergence, which indicates that the current action recognition model has reached the optimal state, and the corresponding action recognition result is the best recognition result.

[0064] The beneficial effects of this invention are as follows: This invention captures and analyzes the movements of personnel on the production line in real time using a camera, effectively monitoring workers' operational behavior. When non-standard movements are detected, timely warnings can be issued, reducing the possibility of human error and equipment damage. By combining convolutional neural networks and long short-term memory networks to construct an action recognition model, accurate recognition of complex movements is achieved, effectively improving the accuracy of recognition, especially for complex and continuous movements. The recognition results are optimized and trained using a feature fusion algorithm, improving the model's applicability and generalization ability on different production lines, adapting to different working conditions and movements. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0066] Figure 1 This is a schematic diagram of the overall method steps of the present invention, which is a method for recognizing the actions of production line personnel based on video understanding algorithms. Detailed Implementation

[0067] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0068] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0069] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0070] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0071] Furthermore, it should be noted in the description of this invention that the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0072] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0073] Example 1

[0074] Reference Figure 1 As an embodiment of the present invention, a method for recognizing the actions of production line personnel based on a video understanding algorithm is provided, comprising the following steps:

[0075] S1: Use cameras to capture the continuous movements of production line workers and extract features from the captured continuous movements.

[0076] Specifically, the method of capturing the continuous movements of production line workers using cameras involves using cameras deployed around the production line to capture real-time video of the workers' continuous movements. The captured continuous movement videos are then organized into a time-series set M, satisfying the formula M = (M1, M2, ..., M...). t ), where V t This represents the action video at time t, where t represents the time node corresponding to the current video.

[0077] The feature extraction of the captured continuous motion is achieved by analyzing pixel motion changes in consecutive video frames, capturing the motion trajectory of the person, and tracking the motion of each pixel in the video between adjacent frames to generate an optical flow field containing direction and velocity information, thereby extracting the person's motion trajectory. The specific implementation is as follows:

[0078] Before extracting action features, the video is decomposed into frames, and key information is extracted from each frame, satisfying the formula M={I1,I2,..,I... n}, where I n This represents the nth frame of the video, where n represents the total number of frames in the current video.

[0079] Based on the frame information from the video decomposition, the human motion features are extracted, as follows:

[0080] Based on the property that the pixel intensity of an object remains constant over time during motion, constraining the pixel intensity in the frame information yields the following:

[0081] I(x,y,t)=I(x+Δx,y+Δy,t+Δt)

[0082] Where I(x,y,t) represents the pixel brightness at position (x,y) at time t, Δx and Δy represent the spatial displacement of the pixel, and Δt represents the time interval;

[0083] If we perform shift decomposition on the information of adjacent frames, then we have:

[0084]

[0085] Where u and v represent the velocity components of the pixel in the x and y directions, respectively. These represent the gradients of the image in the spatial dimension. This represents the gradient of the image over time.

[0086] By constructing an optimized objective function to extract the motion information of each pixel, we have:

[0087]

[0088] Where u and v represent the velocity components of the pixel in the x and y directions, respectively. These represent the gradients of the image in the spatial dimension. This represents the gradient of the image over time. The ratios represent the smoothness of the pixel's velocity change in the x and y directions, respectively, and λ represents the smoothing term coefficient, which is used to adjust the balance between the data terms and the smoothing term.

[0089] By iteratively optimizing the objective function until all video frames at all time points have been iterated, the velocity information u(x,y) and v(x,y) of each pixel in the video can be obtained.

[0090] Based on the velocity information u(x,y) and v(x,y) of each pixel, the amplitude and direction of the motion are calculated, and then...

[0091] Calculation of motion amplitude

[0092]

[0093] Calculation of direction of motion

[0094]

[0095] Where ||V(x,y)|| represents the motion amplitude of the pixel, is the magnitude of the motion speed, and θ(x,y) represents the calculated motion direction;

[0096] Based on the calculated speed and direction of motion, each frame of video data (I) can be extracted. n The motion characteristics in the data are used to form a time series {V1,V2,...,V...} T}, where V T This represents the motion characteristics of the T-th frame.

[0097] S2: Construct an action recognition model based on video processing algorithms to perform action recognition on the extracted motion features.

[0098] Specifically, the action recognition model based on video understanding algorithm is constructed by using convolutional neural network algorithm to perform convolution operation on the frame sequence of video, and using long short-term memory network to capture the time dependence of extracted action features. Finally, the two calculation results are combined to construct the action recognition model.

[0099] Furthermore, the convolution operation on the video frame sequence using the convolutional neural network algorithm involves performing convolution operations on the input video frame sequence in both spatial and temporal dimensions to extract the spatial features and temporal evolution features of the action at each moment, as detailed below:

[0100] The input video frame sequence M = {I1, I2, ..., I...} n}, where I n This represents the nth frame of the video, where n represents the total number of frames in the current video.

[0101] By using convolutional neural networks to perform convolution operations in both spatial and temporal dimensions, we have:

[0102]

[0103] Among them, h t+1 (x,y) represents the convolution output value at time t+1, σ represents the activation function, which is set by the implementer according to the actual application scenario, w(i,j,t) represents the weights of the neural network convolution kernel, and i, j, and t represent the spatial and temporal indices of the convolution kernel parameters, respectively. t(x+i,y+j) represents the pixel value at coordinates (x+i,y+j) in the t-th frame of the input video frame sequence M. b represents the pixel intensity of the original video frame after spatial and temporal displacement. k represents the time step size, which is the size of the convolution kernel in the time dimension. r represents the spatial step size, which is the size of the convolution kernel in the spatial dimension.

[0104] Furthermore, the capture of the temporal dependence of extracted action features using a Long Short-Term Memory (LSTM) network involves employing an LSM network algorithm to capture the temporal dependence within the extracted continuous action features, as detailed below:

[0105] The extracted action feature sequence {V1,V2,...,V T As input to the Long Short-Term Memory network, x is then obtained. t =V t , where x t This represents the amount of input to the Long Short-Term Memory (LSTM) network;

[0106] Based on the input quantity x t and the output h of the convolutional neural network t-1 Given (x, y), calculate the output of the forget gate, then we have:

[0107] f t =σ(W f ·[h t-1 (x,y),x t ]+b f )

[0108] Among them, f t W represents the output of the forget gate at the current moment. f The weight matrix h represents the weight of the forget gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. f σ represents the bias term of the forget gate, and σ represents the activation function, which is set by the implementer according to the actual application scenario.

[0109] By updating memory units using input gates and candidate memories, we have:

[0110] i t =σ(W i ·[h t-1 (x,y),x t ]+b i )

[0111]

[0112] Among them, i tW represents the output result of the input gate. i h represents the weight matrix of the input gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. i σ represents the bias term of the input gate, and σ represents the activation function, which is set by the implementer according to the actual application scenario. W represents the candidate memory value at the current moment. C Let b represent the weight matrix of the candidate memory, tanh represent the hyperbolic tangent function, and b represent the weight matrix of the candidate memory. C Bias terms representing candidate memories;

[0113] Updating the memory unit results in:

[0114]

[0115] Among them, C t f represents the memory unit at the current moment. t C represents the output of the forget gate at the current moment. t-1 i represents the memory unit from the previous time step. t This indicates the input gate's output result. This represents the candidate memory value at the current moment;

[0116] Based on the updated memory units, the output result is calculated, and then we have:

[0117] O t =tanh(C t )*σ(W O ·[h t-1 (x,y),x t ]+b O )

[0118] Among them, O t W represents the output result of the calculation. O h represents the weight matrix of the output gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. O This represents the output bias term, and σ represents the activation function, which is set by the implementer according to the actual application scenario.

[0119] Based on the calculated output, the actions are classified, and then...

[0120] y = Softmax(W o ·h t (x,y)+b O )

[0121] Among them, W O h represents the weight matrix of the output gate. t (x,y) represents the output of the neural network at the current time step, b O This represents the output bias term, where y represents the classification and recognition result corresponding to the action feature.

[0122] S3: Optimize and train the action recognition results based on the feature fusion algorithm.

[0123] Specifically, the optimization training of action recognition results based on feature fusion algorithm is achieved by calculating the probability of various actions and using a loss function to optimize the calculated probabilities until the best optimization effect is achieved.

[0124] Furthermore, the calculation of the probability of each type of action is based on the action recognition results, by statistically analyzing the probabilities of each type of action.

[0125]

[0126] Among them, W c b represents the weight coefficient for action category c in action recognition. c Let y denote the bias term, C denote the total number of categories in action recognition, x denote the input action feature, and P(y = c|x) denote the probability of category c in action recognition.

[0127] Furthermore, the optimization of the calculated probability using the loss function is based on the cross-entropy loss function, as detailed below:

[0128]

[0129] Where P(y=i|x) represents the probability of category i in action recognition, i represents the index of action recognition type, C represents the total number of categories in action recognition, L represents the output of the loss function, and L is the difference between the output of the recognition model and the true label.

[0130] It should be noted that the optimization of the loss function involves repeatedly calculating the output value of the loss function until the output value converges to a stable low value. Then, we have...

[0131] When the output of the loss function satisfies |L t+1 -L t When |→0, it means that the current loss function has reached maximum convergence, which indicates that the current action recognition model has reached the optimal state, and the corresponding action recognition result is the best recognition result.

[0132] Furthermore, if the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer) to...

[0133] Servers or network devices, etc., execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0135] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0136] Furthermore, in order to provide a concise description of exemplary embodiments, not all features of actual embodiments (i.e., those features that are not relevant to the currently considered best mode for carrying out the invention, or those features that are not relevant to implementing the invention) may be omitted.

[0137] It should be understood that numerous specific implementation decisions can be made during the development of any practical implementation, such as in any engineering or design project. Such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, the development effort will be a routine work of design, manufacturing, and production without requiring much experimentation.

[0138] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for recognizing human actions on a production line based on a video understanding algorithm, characterized in that: Includes the following steps, The system uses cameras to capture the continuous movements of production line workers and extracts features from the captured movements. An action recognition model is constructed based on video processing algorithms to perform action recognition on the extracted motion features, specifically as follows: The convolutional neural network algorithm is used to perform convolution operations on the frame sequence of the video, and the long short-term memory network is used to capture the temporal dependence of the extracted action features. Finally, the two calculation results are combined to construct an action recognition model. The action recognition results are optimized and trained based on a feature fusion algorithm, specifically as follows: Based on the action recognition results, the probability of each type of action is calculated, and the loss function is used to optimize the training based on the calculated probability results.

2. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 1, characterized in that: The specific steps for feature extraction from the captured continuous actions are as follows: Based on the frame information from the video decomposition, the human motion features are extracted, as follows: Based on the property that the pixel intensity of an object remains constant over time during motion, constraining the pixel intensity in the frame information yields the following: I(x,y,t)=I(x+Δx,y+Δy,t+Δt) Where I(x,y,t) represents the pixel brightness at position (x,y) at time t, Δx and Δy represent the spatial displacement of the pixel, and Δt represents the time interval; If we perform displacement decomposition on the information of adjacent frames, then we have: Where u and v represent the velocity components of the pixel in the x and y directions, respectively. These represent the gradients of the image in the spatial dimension. This represents the gradient of the image over time.

3. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 2, characterized in that: By constructing an optimized objective function to extract the motion information of each pixel, we have: Where u and v represent the velocity components of the pixel in the x and y directions, respectively. These represent the gradients of the image in the spatial dimension. It represents the gradient of the image in the time dimension, ▽u and ▽v respectively represent the smoothness of the velocity change of the pixel in the x and y directions, and λ represents the smoothing term coefficient, which is used to adjust the balance between the data terms and the smoothing term; By iteratively optimizing the objective function until all video frames at all times have been iterated, the velocity information u(x,y) and v(x,y) of each pixel in the video can be obtained.

4. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 3, characterized in that: Based on the velocity information u(x,y) and v(x,y) of each pixel, the amplitude and direction of the motion are calculated, and then... Calculation of motion amplitude Calculation of direction of motion Where ||V(x,y)|| represents the motion amplitude of the pixel, is the magnitude of the motion speed, and θ(x,y) represents the calculated motion direction; Based on the calculated speed and direction of motion, each frame of video data (I) can be extracted. n The motion characteristics in the data are used to form a time series {V1,V2,...,V...} T }, where V T This represents the motion characteristics of the T-th frame.

5. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 4, characterized in that: The specific steps of performing convolutional operations on the video frame sequence using the convolutional neural network algorithm are as follows: The input video frame sequence M = {I1, I2, ..., I...} n }, where I n This represents the nth frame of the video, where n represents the total number of frames in the current video. By using convolutional neural networks to perform convolution operations in both spatial and temporal dimensions, we have: Among them, h t+1 (x,y) represents the convolution output value at time t+1, σ represents the activation function, which is set by the implementer according to the actual application scenario, w(i,j,t) represents the weights of the neural network convolution kernel, and i, j, and t represent the spatial and temporal indices of the convolution kernel parameters, respectively. t (x+i,y+j) represents the pixel value at coordinates (x+i,y+j) in the t-th frame of the input video frame sequence M. b represents the pixel intensity of the original video frame after spatial and temporal displacement. k represents the time step size, which is the size of the convolution kernel in the time dimension. r represents the spatial step size, which is the size of the convolution kernel in the spatial dimension.

6. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 5, characterized in that: The extraction of action features is captured using a Long Short-Term Memory (LSTM) network to capture time-dependent aspects, as detailed below: The extracted action feature sequence {V1,V2,...,V T As input to the Long Short-Term Memory network, x is then obtained. t =V t , where x t This represents the amount of input to the Long Short-Term Memory (LSTM) network; Based on input quantity x t and the output h of the convolutional neural network t-1 Given (x, y), calculate the output of the forget gate, then we have: f t =σ(W f ·[h t-1 (x,y),x t ]+b f ) Among them, f t W represents the output of the forget gate at the current moment. f h represents the weight matrix of the forget gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. f σ represents the bias term of the forget gate, and σ represents the activation function, which is set by the implementer according to the actual application scenario. By updating memory units using input gates and candidate memories, we have: i t =σ(W i ·[h t-1 (x,y),x t ]+b i ) Among them, i t W represents the output result of the input gate. i h represents the weight matrix of the input gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. i σ represents the bias term of the input gate, and σ represents the activation function, which is set by the implementer according to the actual application scenario. W represents the candidate memory value at the current moment. C Let b represent the weight matrix of the candidate memory, tanh represent the hyperbolic tangent function, and b represent the weight matrix of the candidate memory. C Bias terms representing candidate memories; Updating the memory unit results in: Among them, C t f represents the memory unit at the current moment. t C represents the output of the forget gate at the current moment. t-1 i represents the memory unit from the previous time step. t This indicates the input gate's output result. This represents the candidate memory value at the current moment; Based on the updated memory units, the output result is calculated, and then we have: O t =tanh(C t )*σ(W O ·[h t-1 (x,y),x t ]+b O ) Among them, O t W represents the output result of the calculation. O h represents the weight matrix of the output gate. t-1 (x,y) represents the output of the neural network at the previous time step, x t b represents the action characteristics input at the current moment. O This represents the output bias term, and σ represents the activation function, which is set by the implementer according to the actual application scenario. Based on the calculated output, the actions are classified, and then... y=Softmax(W o ·h t (x,y)+b O ) Among them, W O h represents the weight matrix of the output gate. t (x,y) represents the output of the neural network at the current time step, b O This represents the output bias term, where y represents the classification and recognition result corresponding to the action feature.

7. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 6, characterized in that: The specific calculation of the probability of each type of action is as follows: Based on the action recognition results, the probabilities of various actions are calculated, and then we have: Among them, W c b represents the weight coefficient for action category c in action recognition. c Let y denote the bias term, C denote the total number of categories in action recognition, x denote the input action feature, and P(y = cx) denote the probability of category c in action recognition.

8. The method for recognizing production line personnel actions based on video understanding algorithms as described in claim 7, characterized in that, The optimization training using the loss function is described in detail below: Where P(y=ix) represents the probability of category i in action recognition, i represents the index of action recognition type, C represents the total number of categories in action recognition, L represents the output of the loss function, and L is the difference between the output of the recognition model and the true label; Optimizing the loss function involves repeatedly calculating its output value until it converges to a stable low value. Then, we have... When the output of the loss function satisfies |L t+1 -L t When |→0, it means that the current loss function has reached maximum convergence, which indicates that the current action recognition model has reached the optimal state, and the corresponding action recognition result is the best recognition result.

Images