Force estimation method of crane pipe connection firmness confirmation action and related product
The GT-ForceNet model, which combines graph convolutional networks and LSTM networks, solves the problem of subjective judgment in confirming the firmness of loading arms during oil and gas tanker filling operations, and achieves precise quantification of pulling force, thereby improving operational safety and reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, the confirmation of the tightness of the loading arm connection during oil and gas tanker filling operations relies on the subjective judgment of workers, lacks objective quantitative standards, poses a risk of misjudgment, and cannot provide real-time feedback on the tightness of the connection, leading to safety hazards.
The GT-ForceNet model, which combines graph convolutional networks and LSTM networks, is used to analyze the force and type of a worker's pulling action by acquiring time series images of the target region. It provides a quantitative force estimation method, including human posture estimation and the application of pre-trained models.
It enables precise quantification of the force during the pulling process, significantly improving the safety and reliability of the operation and reducing safety hazards such as oil and gas leaks and fires.
Smart Images
Figure CN122023879A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of oil and gas tanker filling technology, and more specifically to a method for estimating the force of an action to confirm the firmness of an arm-load connection and related products. Background Technology
[0002] Currently, during oil and gas tanker filling operations, workers typically test the connection strength between the loading arm and the tanker by manually pulling on it. However, this method relies entirely on the worker's personal experience and subjective judgment, lacking objective and quantitative strength assessment standards. This not only carries the risk of misjudgment, but also fails to provide real-time and accurate feedback on the actual tightness of the connection, thus creating significant safety hazards such as oil and gas leaks and fires. Therefore, how to provide a force estimation method for confirming the stability of the loading arm connection, which can accurately quantify the force applied during the pulling process to significantly improve the safety and reliability of the operation, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0003] In view of the above problems, the present invention is proposed to provide a force estimation method and related products for confirming the stability of loading arm connection to overcome or at least partially solve the above problems.
[0004] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a method for estimating the force of an action to confirm the stability of a loading arm connection is provided, comprising the following steps: S1: Obtain the time series of the target region image; wherein, the time series of the target region image includes N frames of the target region image; S2: Input the time series of the target area image into the human pose estimation model for processing to obtain the time series of human skeleton data; S3: Input the time series of the human skeleton data into the pre-trained GT-ForceNet model to obtain the pulling force and pulling type of the worker in each frame of the target region image; wherein, the GT-ForceNet model includes a two-layer graph convolutional network, a splicing layer, a Flatten layer, an LSTM network, a fully connected layer group and a softmax layer connected in sequence; the fully connected layer group includes a first fully connected layer and a second fully connected layer in parallel.
[0005] Preferably, S3 specifically includes the following steps: S31: The time series data of the human skeleton { The input is fed into the two-layer graph convolutional network to obtain the high-level spatial feature time series. };in, The time series of the human skeletal data { The nth frame of human skeleton data in}; The time series representing the high-level spatial features { High-level spatial features of the nth frame in}; ; S32: High-rise space characteristics Elbow-shoulder velocity matrix Elbow-shoulder acceleration matrix Angular velocity matrix of elbow-shoulder The input is fed into the stitching layer for stitching operation to obtain stitching features. ; S33: Concatenate the feature time series The image is processed sequentially through a Flatten layer, an LSTM network, and the first fully connected layer to obtain the pulling force of the worker in each frame of the target region image. splicing feature time series The image is processed sequentially through a Flatten layer, an LSTM network, a second fully connected layer, and a softmax layer to obtain the type of pulling or dragging by the worker in each frame of the target region image. in, This represents the spliced feature time series. The splicing features of the nth frame in the image.
[0006] Preferably, S31 is implemented based on the following formula: When n=1: ; ); when hour: ; )); in, Representing low-level spatial feature time series { The low-level spatial features of the nth frame in}; Representing low-level spatial feature time series { The low-level spatial features of the (n-1)th frame in}; This represents a graph convolutional network function; Indicates a splicing operation; This represents the adjacency matrix of a predefined human skeleton diagram.
[0007] Preferably, S31 is implemented based on the following formula: when hour: ; )); When n=N: ; ); in, Representing low-level spatial feature time series { The low-level spatial features of the nth frame in}; Representing low-level spatial feature time series { The low-level spatial features of the (n+1)th frame in}; This represents a graph convolutional network function; Indicates a splicing operation; This represents the adjacency matrix of a predefined human skeleton diagram.
[0008] Preferably, the above-mentioned force estimation method further includes the following steps: S4: Optimize and adjust the pulling force of the worker in each frame of the target area image to obtain the optimized pulling force of the worker in each frame of the target area image.
[0009] Preferably, S4 is implemented based on the following formula: ×( ); In the formula, This represents the confidence level of the pulling force of the worker in the target region image of the i-th frame; when hour, The value is 0; This represents the pulling force of the worker in the nth frame target region image obtained by S33; This represents the optimized pulling force of the worker in the nth frame target region image obtained by S4.
[0010] In a second aspect, a force estimation system for confirming the stability of a loading arm connection is provided, which is used to implement the force estimation method for confirming the stability of a loading arm connection as described in the first aspect, including an acquisition module, a human posture estimation model and a pre-trained GT-ForceNet model. The acquisition module is used to acquire a time series of images of the target region; wherein, the time series of images of the target region includes N frames of images of the target region; The human pose estimation model is used to extract the skeletal data of the time series of the target region image to obtain the human skeletal data time series. The pre-trained GT-ForceNet model is used to process the time series of human skeleton data to obtain the pulling force and pulling type of the worker in each frame of the target region image; wherein, the GT-ForceNet model includes a two-layer graph convolutional network, a splicing layer, a Flatten layer, an LSTM network and a fully connected layer connected in sequence.
[0011] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the force estimation method for confirming the robustness of the loading arm connection as described in the first aspect.
[0012] Fourthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the force estimation method for confirming the robustness of the loading arm connection as described in the first aspect.
[0013] Fifthly, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the force estimation method for confirming the robustness of the loading arm connection as described in the first aspect is implemented.
[0014] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for estimating the force of the action to confirm the firmness of the loading arm connection and related products, which can accurately quantify the force applied during the pulling process, and can significantly improve the safety and reliability of the operation. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0016] Figure 1 This is a flowchart of a force estimation method for confirming the stability of an arm-load connection provided in an embodiment of the present invention; Figure 2 This is a tanker truck filling scenario diagram provided in an embodiment of the present invention; Figure 3 This is a kinematic analysis diagram of the pulling action provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Firstly, such as Figures 1-2 As shown in the figure, an embodiment of the present invention discloses a method for estimating the force of an action to confirm the firmness of an arm-load connection, comprising the following steps: S1: Obtain the time series of the target region image; wherein, the time series of the target region image includes N frames of the target region image; In one embodiment, N=60; In one embodiment, the time series of the target region image is obtained based on the following steps; 1) Use a camera to acquire video streams; It should be noted that the camera frame rate is 230fps.
[0019] 2) After decoding and frame extraction of the video stream, the Retinex algorithm is applied to correct non-uniform illumination; It should be noted that the frame rate is 30fps. A lightweight preprocessing module is deployed in the camera to enable localized video stream decoding and frame extraction. Specifically, the lightweight preprocessing module is deployed inside each camera or on the embedded device directly connected to it (such as Jetson Nano, Raspberry Pi + AI accelerator stick) to enable localized video stream decoding and frame extraction.
[0020] After correction, dark areas in the image due to insufficient light will be brightened, and details in overexposed areas due to reflections will be restored, resulting in a more uniform visual effect and clearer details throughout the image.
[0021] 3) If five key targets—worker, loading arm, tanker, tank body, and pipeline connection—are simultaneously detected stably in N consecutive frames of the corrected image, then these N consecutive frames are taken as the time series of the target area image.
[0022] S2: Input the time series of the target area image into the human pose estimation model for processing to obtain the time series of human skeleton data; S3: Input the time series of the human skeleton data into the pre-trained GT-ForceNet model to obtain the pulling force and pulling type of the worker in each frame of the target region image; wherein, the GT-ForceNet model includes a two-layer graph convolutional network, a splicing layer, a Flatten layer, an LSTM network, a fully connected layer group and a softmax layer connected in sequence; the fully connected layer group includes a first fully connected layer and a second fully connected layer in parallel.
[0023] The pre-trained GT-ForceNet model is obtained based on the following steps: 1) Obtain the dataset: Fifteen subjects of different body types were recruited to perform five different pulling actions (left-handed pulling, right-handed pulling, two-handed pulling, reverse pulling, and standard action). Each action was repeated three times. A four-directional high frame rate camera system (200 FPS) was used to simultaneously acquire video data from multiple perspectives. A force sensor was integrated into the operating device to achieve millisecond-level synchronization between the pulling force and visual data. The final dataset included 7,000 pulling action samples (including 1,000 interference actions (i.e., left-handed pulling, right-handed pulling, two-handed pulling, and reverse pulling)). Each sample consisted of N frames of COCO format skeleton data time series. Each frame of skeleton data was associated with one real force (force sensor range 0-500N, accuracy ±0.5%FS). Each sample was labeled with the pulling type (i.e., left-handed pulling, right-handed pulling, two-handed pulling, reverse pulling, or standard action).
[0024] 2) Train the GT-ForceNet model using the dataset (divided into training and validation sets) to obtain the pre-trained GT-ForceNet model.
[0025] Understandably, given the characteristics of the dataset, this invention employs a structured training strategy: setting the batch size to 64 and the maximum training epochs to 100. To optimize model generalization performance, this invention implements a dynamic early stopping mechanism—calculating the validation set loss function value after each training epoch, and immediately terminating training when the loss decreases less than a threshold for 10 consecutive epochs. This strategy, based on the second derivative variation characteristics of the validation loss curve, effectively suppresses model overfitting while ensuring that the parameters converge to Pareto optimality.
[0026] In one embodiment, S3 specifically includes the following steps: S31: The time series data of the human skeleton { The input is fed into the two-layer graph convolutional network to obtain the high-level spatial feature time series. };in, The time series of the human skeletal data { The nth frame of human skeleton data in}; The time series representing the high-level spatial features { High-level spatial features of the nth frame in}; ; It is understandable that: { }= , ,..., ; { }= , ,..., ; In one embodiment, S31 is implemented based on the following formula: When n=1: ; ); when hour: ; )); in, Representing low-level spatial feature time series { The low-level spatial features of the nth frame in}; Representing low-level spatial feature time series { The low-level spatial features of the (n-1)th frame in}; This represents a graph convolutional network function; Indicates a splicing operation; This represents the adjacency matrix of a predefined human skeleton diagram.
[0027] It is understandable that: { }= , ,..., .
[0028] In one embodiment, S31 is implemented based on the following formula: when hour: ; )); When n=N: ; ); in, Representing low-level spatial feature time series { The low-level spatial features of the nth frame in}; Representing low-level spatial feature time series { The low-level spatial features of the (n+1)th frame in}; This represents a graph convolutional network function; Indicates a splicing operation; This represents the adjacency matrix of a predefined human skeleton diagram.
[0029] S32: High-rise space characteristics Elbow-shoulder velocity matrix Elbow-shoulder acceleration matrix Angular velocity matrix of elbow-shoulder The input is fed into the stitching layer for stitching operation to obtain stitching features. ; The specific expression is: = , , , ); This represents the elbow-shoulder velocity matrix corresponding to the human skeleton data in the nth frame; This represents the elbow-shoulder acceleration matrix corresponding to the nth frame of human skeletal data; This represents the elbow-shoulder angular velocity matrix corresponding to the human skeleton data in the nth frame; Using the time series of the human skeletal data { Calculate and obtain the elbow-shoulder velocity matrix. Elbow-shoulder acceleration matrix Angular velocity matrix of elbow-shoulder (This part is prior art); For example: velocity = joint displacement / interval time. The elbow-shoulder velocity matrix can be calculated by the changes in joint coordinates between adjacent frames. .
[0030] Specifically: During the pulling motion, the shoulder and elbow are key areas for skeletal force generation. Based on the principle of inverse kinematics, the kinematic parameters of the shoulder and elbow (i.e., the elbow-shoulder velocity matrix) can be extracted. Elbow-shoulder acceleration matrix Angular velocity matrix of elbow-shoulder ), which is crucial for strength estimation.
[0031] like Figure 3 As shown, pink dots represent the keypoint positions in frame t-1, connected by dashed lines, while red dots represent the keypoint positions in frame t, connected by solid lines. This represents the velocity of the key point in frame t. Represents the angle of the key point in frame t ( (2, ... N). This invention integrates the velocity, acceleration, and angular velocity of the shoulder and elbow to enhance spatial dynamic characteristics.
[0032] S33: Concatenate the feature time series The image is processed sequentially through a Flatten layer, an LSTM network, and the first fully connected layer to obtain the pulling force of the worker in each frame of the target region image. splicing feature time series The image is processed sequentially through a Flatten layer, an LSTM network, a second fully connected layer, and a softmax layer to obtain the type of pulling or dragging by the worker in each frame of the target region image. in, This represents the spliced feature time series. The splicing features of the nth frame in the image.
[0033] It is understandable that for the type output task, a softmax processing layer is needed after the fully connected layer. For the force estimation task, no softmax processing layer is needed after the fully connected layer. These two tasks correspond to their respective fully connected layers. In this invention, the pulling force is output by the first fully connected layer; the pulling type is output by the softmax layer.
[0034] In one embodiment, the above-described force estimation method further includes the following steps: S4: Optimize and adjust the pulling force of the worker in each frame of the target area image to obtain the optimized pulling force of the worker in each frame of the target area image.
[0035] In one embodiment, S4 is specifically implemented based on the following formula: ×( ); In the formula, This represents the confidence level of the pulling force of the worker in the target region image of the i-th frame; when hour, The value is 0; This represents the pulling force of the worker in the nth frame target region image obtained by S33; This represents the optimized pulling force of the worker in the nth frame target region image obtained by S4.
[0036] It is understood that the optimized pulling force obtained by using the present invention can be used to provide corresponding early warning in practice to determine whether the arm arm connection firmness confirmation action (i.e., pulling action) is qualified. Specifically: If the time series of pulling force is optimized If there is an optimized pulling force greater than 150, a Level I alarm will be triggered (specifically, an audible and visual alarm will be triggered), and the loading arm filling will be stopped immediately. }= , ,..., ; If the time series of pulling force is optimized If all optimized pulling forces are below 50N, a Level II alarm will be triggered, and a voice prompt will remind you that the pulling force is too weak.
[0037] It is understandable that: }= , ,..., ; In a second aspect, embodiments of the present invention also provide a force estimation system for confirming the stability of the loading arm connection, used to implement the force estimation method for confirming the stability of the loading arm connection described in the first aspect, including an acquisition module, a human posture estimation model, and a pre-trained GT-ForceNet model; The acquisition module is used to acquire a time series of images of the target region; wherein, the time series of images of the target region includes N frames of images of the target region; The human pose estimation model is used to extract the skeletal data of the time series of the target region image to obtain the human skeletal data time series. The pre-trained GT-ForceNet model is used to process the time series of human skeleton data to obtain the pulling force and pulling type of the worker in each frame of the target region image; wherein, the GT-ForceNet model includes a two-layer graph convolutional network, a splicing layer, a Flatten layer, an LSTM network and a fully connected layer connected in sequence.
[0038] Thirdly, embodiments of the present invention also provide an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 401, a communications interface 402, a memory 403, and a communication bus 404. The processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions from the memory 403 to execute a force estimation method for verifying the stability of the loading arm connection.
[0039] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0040] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the force estimation method for confirming the robustness of the loading arm connection provided by the above methods.
[0041] Fifthly, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a force estimation method for performing the arm connection firmness confirmation action provided by the methods described above.
[0042] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0043] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0044] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0045] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for estimating the force of an action to confirm the firmness of an arm-load connection, characterized in that, Includes the following steps: S1: Obtain the time series of the target region image; wherein, the time series of the target region image includes N frames of the target region image; S2: Input the time series of the target area image into the human pose estimation model for processing to obtain the time series of human skeleton data; S3: Input the time series of the human skeleton data into the pre-trained GT-ForceNet model to obtain the pulling force and pulling type of the worker in each frame of the target region image; wherein, the GT-ForceNet model includes a two-layer graph convolutional network, a splicing layer, a Flatten layer, an LSTM network, a fully connected layer group and a softmax layer connected in sequence; the fully connected layer group includes a first fully connected layer and a second fully connected layer in parallel.
2. The method for estimating the force of the action to confirm the firmness of the loading arm connection according to claim 1, characterized in that, S3 specifically includes the following steps: S31: The time series data of the human skeleton { The input is fed into the two-layer graph convolutional network to obtain the high-level spatial feature time series. };in, The time series of the human skeletal data { The nth frame of human skeleton data in}; The time series representing the high-level spatial features { High-level spatial features of the nth frame in}; ; S32: High-rise space characteristics Elbow-shoulder velocity matrix Elbow-shoulder acceleration matrix Angular velocity matrix of elbow-shoulder The input is fed into the stitching layer for stitching operation to obtain stitching features. ; S33: Concatenate the feature time series The image is processed sequentially through a Flatten layer, an LSTM network, and the first fully connected layer to obtain the pulling force of the worker in each frame of the target region image. splicing feature time series The image is processed sequentially through a Flatten layer, an LSTM network, a second fully connected layer, and a softmax layer to obtain the type of pulling or dragging by the worker in each frame of the target region image. in, This represents the spliced feature time series. The splicing features of the nth frame in the image.
3. The method for estimating the force of the action to confirm the firmness of the loading arm connection according to claim 2, characterized in that, S31 is implemented based on the following formula: When n=1: ; ); when hour: ; )); in, Represents the time series of low-level spatial features { The low-level spatial features of the nth frame in}; Representing low-level spatial feature time series { The low-level spatial features of the (n-1)th frame in}; This represents a graph convolutional network function; Indicates a splicing operation; This represents the adjacency matrix of a predefined human skeleton diagram.
4. The method for estimating the force of the action to confirm the firmness of the loading arm connection according to claim 2, characterized in that, S31 is implemented based on the following formula: when hour: ; )); When n=N: ; ); in, Representing low-level spatial feature time series { The low-level spatial features of the nth frame in}; Representing low-level spatial feature time series { The low-level spatial features of the (n+1)th frame in}; This represents a graph convolutional network function; Indicates a splicing operation; This represents the adjacency matrix of a predefined human skeleton diagram.
5. The method for estimating the force of the action to confirm the firmness of the loading arm connection according to claim 1, characterized in that, It also includes the following steps: S4: Optimize and adjust the pulling force of the worker in each frame of the target area image to obtain the optimized pulling force of the worker in each frame of the target area image.
6. The method for estimating the force of the action to confirm the firmness of the loading arm connection according to claim 5, characterized in that, S4 is specifically implemented based on the following formula: ×（ ）； In the formula, This represents the confidence level of the pulling force of the worker in the target region image of the i-th frame; when hour, The value is 0; This represents the pulling force of the worker in the nth frame target region image obtained by S33; This represents the optimized pulling force of the worker in the nth frame target region image obtained by S4.
7. A force estimation system for confirming the firmness of an arm-load connection, characterized in that, The force estimation method for implementing the loading arm connection firmness confirmation action as described in any one of claims 1-6 includes an acquisition module, a human posture estimation model, and a pre-trained GT-ForceNet model. The acquisition module is used to acquire a time series of images of the target region; wherein, the time series of images of the target region includes N frames of images of the target region; The human pose estimation model is used to extract the skeletal data of the time series of the target region image to obtain the human skeletal data time series. The pre-trained GT-ForceNet model is used to process the time series of human skeleton data to obtain the pulling force and pulling type of the worker in each frame of the target region image; wherein, the GT-ForceNet model includes a two-layer graph convolutional network, a splicing layer, a Flatten layer, an LSTM network and a fully connected layer connected in sequence.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the force estimation method for confirming the secureness of the loading arm connection as described in any one of claims 1 to 6.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the force estimation method for confirming the secure connection of the loading arm as described in any one of claims 1 to 6.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the force estimation method for confirming the secure connection of the loading arm as described in any one of claims 1 to 6.