Method and apparatus for determining operation behavior specification, electronic device, and storage medium

Abstract

The application discloses a method and device for determining operation behavior, electronic equipment and storage medium, relates to the technical field of computer vision, and is used for solving the container falling event. The method comprises the following steps: acquiring a video stream of a work scene, decoding the video stream, obtaining a plurality of image frames, and the work scene comprising a sling container lifting operation and / or a sling container returning operation; based on the plurality of image frames, determining a target number of stoppages and a target stoppage duration of the sling during the operation; and based on the target number of stoppages and the target stoppage duration, determining whether the operation behavior of an operator is standard, the operator being an operator who controls the sling to operate. The embodiment of the application can detect whether the operation behavior of the operator is standard, and reduce the occurrence of the container falling.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device, and storage medium for determining operational behavior norms. Background Technology

[0002] Ports are vital infrastructure for national economic and social development, serving as gateways to the outside world and holding immense significance. Gantry cranes, quay cranes, and other heavy machinery are crucial pieces of equipment in ports. During quay crane operations involving container return or retrieval, containers may suddenly fall from the air. Therefore, preventing container falls is of paramount importance. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and storage medium for determining operational behavior standards, which can detect whether the operator's work behavior is standardized and reduce the probability of container falling incidents.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a method for determining operational behavior specifications, comprising:

[0006] The video stream of the operation scene is acquired and decoded to obtain multiple image frames. The operation scene includes lifting and / or returning of the lifting equipment.

[0007] Based on multiple image frames, determine the number of target pauses and the duration of target pauses during the operation of the spreader;

[0008] Based on the target number of pauses and the target pause duration, it is determined whether the operator's operating behavior is standardized. The operator refers to the person who controls the lifting equipment to carry out the operation.

[0009] In this embodiment of the application, by determining the target number of pauses and the target pause duration during the operation of the spreader, it is possible to determine whether the operator's behavior is standardized. This can improve the accuracy of determining the behavior standard, effectively reduce the occurrence of container falls, prevent container fall incidents from the root, and improve the safety of spreader operations.

[0010] In one possible implementation, determining the target number of pauses and the target pause duration of the spreader during operation based on multiple image frames includes:

[0011] For each image frame, target detection is performed on the image frame based on the first target neural network to obtain the detection result of the image frame. The detection result is used to indicate the position information of the spreader and / or the truck.

[0012] Based on the detection results of the first group of image frames in multiple image frames, it is determined whether the spreader has entered the working state. The first group of image frames includes the first target image frame and all image frames before the first target image frame.

[0013] When the spreader enters the working state, based on the detection results of the second group of image frames in multiple image frames, it is determined whether the current operation has ended. The second group of image frames includes the second target image frame and all image frames before the second target image frame. The second target image frame is the image frame after the first target image frame.

[0014] If the current task is determined to be completed, the number of target pauses and the duration of target pauses are determined based on the detection results of multiple image frames.

[0015] In this embodiment, multiple image frames are input into a first target neural network for target detection to obtain the detection results of multiple image frames. Based on the detection results of the first set of image frames, it is determined whether the spreader has entered the working state. After determining that the spreader has entered the working state, it is determined whether the current operation has ended based on the detection results of the second set of image frames. If the current operation has ended, the number of target pauses and the duration of target pauses are determined based on the detection results of multiple image frames. In this way, the accuracy of determining the number of target pauses and the duration of target pauses can be improved, further improving the accuracy of determining the behavior norms and effectively reducing the probability of container falling.

[0016] In one possible implementation, determining whether the lifting device has entered the working state based on the detection results of the first group of image frames among multiple image frames includes:

[0017] If the first set of image frames includes both the truck and the spreader, and the position information of the truck has not changed, and the detection result of the first target image frame indicates that the spreader is located directly above the truck, then the spreader is determined to have entered the working state.

[0018] In this embodiment of the application, the presence or absence of a lifting device is determined by determining whether the first set of image frames includes a container truck and a lifting device, whether the position information of the container truck has changed, and whether the lifting device is located directly above the container truck. This allows for accurate determination of whether the lifting device has entered the working state, providing a basis for subsequent inspection work.

[0019] In one possible implementation, determining whether the current task has ended based on the detection results of a second group of image frames among multiple image frames includes:

[0020] Based on the detection results of the second set of image frames, the motion state of the spreader and the motion state of the truck are determined.

[0021] Determine whether the spreader is in a lifting state based on the movement state of the spreader and the truck.

[0022] When the lifting device is in the lifting state, the detection results of the third set of image frames in the second set of image frames determine whether the current operation has ended. The third set of image frames is the last X image frames in the second set of image frames, where X is a preset value.

[0023] In this embodiment, the motion state of the spreader and the truck is determined based on the detection results of the second set of image frames. Then, based on the motion state of the spreader and the truck, it is determined whether the spreader is in a lifting state. If the spreader is in a lifting state, it is determined whether the current operation has ended based on the detection results of the third set of image frames in the second set of image frames. In this way, it is possible to accurately determine whether the current operation has ended, thereby improving the accuracy of determining the number of target pauses and the duration of target pauses.

[0024] In one possible implementation, determining the motion state of the spreader and the truck based on the detection results of the second set of image frames includes:

[0025] Based on the position information of the truck indicated by the detection results of the second set of image frames, the motion state of the truck is determined, including left and right movement or stationary state.

[0026] Based on the detection results of the second target neural network and the second target image frame, the position information of the lifting device is indicated, and the motion state of the lifting device is determined. The motion state of the lifting device includes a stationary state, upward movement, downward movement, or left and right movement.

[0027] In this embodiment, the motion state of the container truck is determined based on its location information, and the motion state of the spreader is determined based on the location information of the second target neural network and the spreader. In this way, the motion states of the spreader and the container truck can be accurately determined, reducing the occurrence of errors in determining the lifting state of the spreader and providing a basis for determining whether the current operation has ended.

[0028] In one possible implementation, determining the motion state of the lifting device based on the position information of the lifting device indicated by the detection results of the second target neural network and the second target image frame includes:

[0029] Based on the position information of the lifting device indicated by the detection results of the second target image frame, the maximum working area of ​​the lifting device is determined;

[0030] Based on the second target neural network, feature extraction is performed on the maximum working area of ​​the lifting device to obtain the feature points of the lifting device in the second target image frame;

[0031] Obtain the feature points of the lifting device in all image frames except the second target image frame in the second group of image frames;

[0032] Based on the feature points of the lifting device in the second set of image frames, the target line segments of all feature points within the maximum working area are determined.

[0033] Based on the target line segment, determine the motion state of the lifting device.

[0034] In this embodiment, features are extracted from the maximum working area of ​​the spreader based on the second target neural network to obtain the feature points of the spreader in the second target image frame. The feature points of the spreader in all image frames before the second target image frame are also obtained. Then, based on the feature points of the spreader in the second set of image frames, the target line segment of all feature points in the maximum working area is determined. Finally, the motion state of the spreader is determined based on the target line segment. In this way, the accuracy of motion state determination can be improved, the occurrence of errors in determining the lifting state of the spreader can be reduced, and a basis is provided for subsequent determination of whether the current operation has ended.

[0035] In one possible implementation, determining whether the spreader is in a lifting state based on the movement state of the spreader and the movement state of the truck includes:

[0036] When the truck is stationary and the spreader is moving upwards, the spreader is determined to be in a lifting state.

[0037] In this embodiment of the application, the lifting device is determined to be in a lifting state only after the movement state of the truck is determined to be stationary and the movement state of the spreader is upward. In this way, it is possible to accurately determine whether the spreader is in a lifting state, thereby improving the accuracy of determining whether the current operation has ended.

[0038] In one possible implementation, determining whether the current task has ended based on the detection results of the third group of image frames in the second group of image frames includes:

[0039] If the detection result of the second target image frame indicates that the second target image frame does not include the truck, or if the detection result of the third set of image frames indicates that the third set of image frames does not include the lifting equipment, the current operation is determined to be terminated.

[0040] In this embodiment of the application, the current operation is determined by determining whether the second target image frame includes a container truck and whether the third set of image frames all include a spreader. In this way, the current operation can be accurately determined, thereby making the number of target pauses and the target pause duration more accurate and reducing the occurrence of container falls.

[0041] In one possible implementation, the method further includes:

[0042] If the current operation has not been completed, the first number of pauses of the spreader and the duration of the first pause are determined based on the motion state of the spreader in the second set of image frames.

[0043] The third target image frame is determined from multiple image frames, and the third target image frame is the next image frame after the second target image frame;

[0044] The second set of image frames and the third target image frame are redefined as a new second set of image frames, and based on the new second set of image frames, it is determined whether the current operation has ended.

[0045] In this embodiment, if the current operation has not ended, it is necessary to determine the first number of pauses and the first pause duration of the spreader based on the movement state of the spreader in the second set of image frames, and then re-determine whether the current operation has ended based on the third target image frame and the second set of image frames. In this way, it can be ensured that the first number of pauses and the first pause duration obtained when the current operation ends are accurate, which improves the accuracy of the determination of the behavior norms, thereby reducing the occurrence of container falls and improving the safety of spreader operation.

[0046] In one possible implementation, determining the number of pauses and the duration of each pause based on the motion state of the lifting device in the second set of image frames includes:

[0047] If the motion state of the lifting device changes from upward movement to a stationary state in the second set of image frames, the first pause count of the lifting device will be increased by 1.

[0048] If the motion state of the lifting device remains stationary in the second set of image frames, the first pause duration of the lifting device is increased by a target duration, which is determined by the frame rate of the video stream.

[0049] In this embodiment of the application, if the motion state of the spreader changes from upward movement to a stationary state, the number of the first pauses of the spreader is increased by 1; if the motion state of the spreader remains stationary, the duration of the first pause of the spreader is increased by the target duration. In this way, the accuracy of the determination of the first pause duration and the first pause duration can be improved, providing a basis for the subsequent determination of the target pause duration and the target pause duration.

[0050] In one possible implementation, the target number of pauses is the first number of pauses corresponding to the end of the current task, and the target pause duration is the first pause duration corresponding to the end of the current task.

[0051] This ensures that the target pause duration and the target pause duration generated at the end of the current task are determined based on the first pause duration and the first pause duration, thereby improving the accuracy of the target pause duration and the determination of the target pause duration, and thus improving the accuracy of the determination of the behavioral norms.

[0052] In one possible implementation, determining whether the operator's behavior is standardized based on the target number of pauses and the target pause duration includes:

[0053] If the target number of pauses in the spreader is greater than or equal to the preset number, and the target pause duration exceeds the preset duration, then the operator's operational behavior guidelines shall be determined; or,

[0054] If the number of target pauses of the spreader is less than the preset number, or if the target pause duration of the spreader does not exceed the preset duration, the operator's operation is determined to be non-standard.

[0055] In this embodiment, the operator's behavior is determined to be standardized by determining whether the target number of pauses of the spreader is less than the preset number and whether the target pause duration of the spreader exceeds the preset duration. This improves the accuracy of behavior standardization and reduces the occurrence of container falls, thereby enhancing the safety of spreader operations.

[0056] In one possible implementation, the method further includes:

[0057] If it is determined that the operator's actions are not in accordance with regulations, a prompt message will be output to remind the operator; and / or,

[0058] If it is determined that the operator's operation is not in accordance with regulations, facial recognition is performed on the operator to obtain the operator's identity information, and the operator's identity information is reported.

[0059] In this embodiment, if the operator's operation is not standardized, a prompt message can be output. This allows the target personnel to clearly understand whether their operation is standardized and facilitates timely adjustment of their operation. If the operator's operation is not standardized, facial recognition can be performed on the operator to obtain their identity information, and this information can be reported. This allows for the recording of personnel with non-standard operation behavior, supervision of operators, and reduction of non-standard operation situations.

[0060] Secondly, embodiments of this disclosure also provide an apparatus for determining operational behavior specifications, comprising:

[0061] The video acquisition module is used to acquire video streams of the work scene and decode the video streams to obtain multiple image frames. The work scene includes lifting and / or returning of the lifting equipment.

[0062] The first determining module is used to determine the number of target pauses and the duration of target pauses during the operation of the spreader based on multiple image frames;

[0063] The second determination module is used to determine whether the operator's operation behavior is standardized based on the target number of pauses and the target pause duration. The operator is the person who controls the lifting equipment to carry out the operation.

[0064] In one possible implementation, the first determining module is specifically used for:

[0065] For each image frame, target detection is performed on the image frame based on the first target neural network to obtain the detection result of the image frame. The detection result is used to indicate the position information of the spreader and / or the truck.

[0066] Based on the detection results of the first group of image frames in multiple image frames, it is determined whether the spreader has entered the working state. The first group of image frames includes the first target image frame and all image frames before the first target image frame.

[0067] When the spreader enters the working state, based on the detection results of the second group of image frames in multiple image frames, it is determined whether the current operation has ended. The second group of image frames includes the second target image frame and all image frames before the second target image frame. The second target image frame is the image frame after the first target image frame.

[0068] If the current task is determined to be completed, the number of target pauses and the duration of target pauses are determined based on the detection results of multiple image frames.

[0069] In one possible implementation, the first determining module is specifically used for:

[0070] If the first set of image frames includes both the truck and the spreader, and the position information of the truck has not changed, and the detection result of the first target image frame indicates that the spreader is located directly above the truck, then the spreader is determined to have entered the working state.

[0071] In one possible implementation, the first determining module is specifically used for:

[0072] Based on the detection results of the second set of image frames, the motion state of the spreader and the motion state of the truck are determined.

[0073] Determine whether the spreader is in a lifting state based on the movement state of the spreader and the truck.

[0074] When the lifting device is in the lifting state, the detection results of the third set of image frames in the second set of image frames determine whether the current operation has ended. The third set of image frames is the last X image frames in the second set of image frames, where X is a preset value.

[0075] In one possible implementation, the first determining module is specifically used for:

[0076] Based on the position information of the truck indicated by the detection results of the second set of image frames, the motion state of the truck is determined, including left and right movement or stationary state.

[0077] Based on the detection results of the second target neural network and the second target image frame, the position information of the lifting device is indicated, and the motion state of the lifting device is determined. The motion state of the lifting device includes a stationary state, upward movement, downward movement, or left and right movement.

[0078] In one possible implementation, the first determining module is specifically used for:

[0079] Based on the position information of the lifting device indicated by the detection results of the second target image frame, the maximum working area of ​​the lifting device is determined;

[0080] Based on the second target neural network, feature extraction is performed on the maximum working area of ​​the lifting device to obtain the feature points of the lifting device in the second target image frame;

[0081] Obtain the feature points of the lifting device in all image frames except the second target image frame in the second group of image frames;

[0082] Based on the feature points of the lifting device in the second set of image frames, the target line segments of all feature points within the maximum working area are determined.

[0083] Based on the target line segment, determine the motion state of the lifting device.

[0084] In one possible implementation, the first determining module is specifically used for:

[0085] When the truck is stationary and the spreader is moving upwards, the spreader is determined to be in a lifting state.

[0086] In one possible implementation, the first determining module is specifically used for:

[0087] If the detection result of the second target image frame indicates that the second target image frame does not include the truck, or if the detection result of the third set of image frames indicates that the third set of image frames does not include the lifting equipment, the current operation is determined to be terminated.

[0088] In one possible implementation, the first determining module is specifically used for:

[0089] If the current operation has not been completed, the first number of pauses of the spreader and the duration of the first pause are determined based on the motion state of the spreader in the second set of image frames.

[0090] The third target image frame is determined from multiple image frames, and the third target image frame is the next image frame after the second target image frame;

[0091] The second set of image frames and the third target image frame are redefined as a new second set of image frames, and based on the new second set of image frames, it is determined whether the current operation has ended.

[0092] In one possible implementation, the first determining module is specifically used for:

[0093] If the motion state of the lifting device changes from upward movement to a stationary state in the second set of image frames, the first pause count of the lifting device will be increased by 1.

[0094] If the motion state of the lifting device remains stationary in the second set of image frames, the first pause duration of the lifting device is increased by a target duration, which is determined by the frame rate of the video stream.

[0095] In one possible implementation, the target number of pauses is the first number of pauses corresponding to the end of the current task, and the target pause duration is the first pause duration corresponding to the end of the current task.

[0096] In one possible implementation, the second determining module is specifically used for:

[0097] If the target number of pauses in the spreader is greater than or equal to the preset number, and the target pause duration exceeds the preset duration, then the operator's operational behavior guidelines shall be determined; or,

[0098] If the number of target pauses of the spreader is less than the preset number, or if the target pause duration of the spreader does not exceed the preset duration, the operator's operation is determined to be non-standard.

[0099] In one possible implementation, the device further includes:

[0100] The information output module is used to output a prompt message when it is determined that the operator's operation is not in accordance with regulations. The prompt message is used to remind the operator; and / or,

[0101] If it is determined that the operator's operation is not in accordance with regulations, facial recognition is performed on the operator to obtain the operator's identity information, and the operator's identity information is reported.

[0102] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the method for determining the operating behavior specifications described in any of the above possible embodiments is executed.

[0103] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the method for determining operational behavior specifications as described in any of the possible implementations above.

[0104] For a detailed description of the second to fourth aspects and their various implementations in this application, please refer to the detailed description in the first aspect and its various implementations; and for a detailed description of the beneficial effects of the second to fourth aspects and their various implementations, please refer to the beneficial effect analysis in the first aspect and its various implementations, which will not be repeated here.

[0105] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0106] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 A flowchart illustrating a method for determining operational behavior specifications provided in an embodiment of this application;

[0108] Figure 2 A flowchart illustrating a method for determining the target number of pauses and the target pause duration, provided in an embodiment of this application;

[0109] Figure 3 A flowchart illustrating a method for determining whether a current job has ended, provided in an embodiment of this application;

[0110] Figure 4 A flowchart illustrating a method for determining the motion state of a lifting device and a container truck, provided in an embodiment of this application;

[0111] Figure 5 A flowchart illustrating a method for determining the motion state of a lifting device, as provided in an embodiment of this application;

[0112] Figure 6This application provides a schematic diagram of the structure of a detection operation behavior standardization system.

[0113] Figure 7 A schematic diagram of the structure of a device for determining operational behavior specifications provided in an embodiment of this application;

[0114] Figure 8 A schematic diagram of the structure of another device for determining operational behavior specifications provided in an embodiment of this application;

[0115] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0116] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0117] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0118] Research has found that existing technologies only prevent people and vehicles from being hit by falling containers by avoiding the presence of people or vehicles under the spreader, thereby reducing the losses caused by accidents. However, they cannot fundamentally prevent containers from falling, and there may be situations where people and vehicles cannot be detected in time, reducing the safety of spreader operations.

[0119] However, after the spreader with container attached places the container onto the truck, the quay crane operator needs to release the latches between the spreader and the container before raising the empty spreader. During this process, the most likely scenario for the container to fall is that the latches between the spreader and the container are not fully released due to a malfunction. The spreader lifts the container along with it, and if the operator does not notice this in time, the container may detach during the lifting process, causing a serious accident. Alternatively, when the empty spreader is attached to the container on the truck, the operator needs to lock the latches between the spreader and the container before raising the spreader with container attached. During this process, the most likely scenario for the container to fall is that the latches between the spreader and the container are not fully locked, causing the container to detach during the lifting process and resulting in a serious accident.

[0120] Therefore, during the operation of the spreader, the stage where container falls are most likely to occur is during the lifting phase of the spreader. The lifting of the spreader is performed by the operators. In other words, checking whether the operators' operating behavior is in accordance with regulations is the key to solving container falling incidents.

[0121] To address the aforementioned issues, this application provides a method for determining operational behavior standards, comprising: acquiring a video stream of a work scenario and decoding the video stream to obtain multiple image frames; the work scenario includes lifting a container with a spreader and / or returning a container with a spreader; based on the multiple image frames, determining the target number of pauses and the target pause duration of the spreader during the work process; and based on the target number of pauses and the target pause duration, determining whether the operator's operational behavior is standardized, wherein the operator is the person controlling the spreader to perform the work.

[0122] In this embodiment, by determining the target number of pauses and the target pause duration during the operation of the spreader, the operator's behavior is determined to be standardized. This improves the accuracy of behavior standard determination, effectively reduces the possibility of containers falling due to improper locking connections between the spreader and the container, prevents container falling incidents from the root, and improves the safety of spreader operations.

[0123] The execution subject of the method for determining operational behavior specifications provided in this application is generally an electronic device, server, or other processing device with a certain computing power. The electronic device can be a mobile device, handheld device, computing device, vehicle-mounted device, wearable device, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms.

[0124] In some possible implementations, the method for determining this operational behavior specification can be achieved by the processor calling computer-readable instructions stored in memory.

[0125] The following describes the application scenarios of the method for determining operational behavior specifications provided in the embodiments of this application.

[0126] The method for determining operational behavior specifications provided in this application embodiment is applied to the lifting equipment operation scenario of quay cranes.

[0127] The method for determining operational behavior specifications provided in other embodiments can also be applied to material handling operation scenarios in any factory. For example, the lifting device can be a robotic arm that handles containers, the container can be any type of container (such as a cardboard box), and the cargo ship and truck can be any vehicle capable of loading containers (such as a truck, train, etc.), without any specific limitation.

[0128] The method for determining operational behavior specifications provided in the embodiments of this application will be described below.

[0129] See Figure 1 The diagram shows a flowchart of a method for determining operational behavior specifications provided in an embodiment of this application. The method includes steps S101 to S103:

[0130] S101, acquire the video stream of the operation scene and decode the video stream to obtain multiple image frames. The operation scene includes lifting the container with a spreader and / or returning the container with a spreader.

[0131] Specifically, video streams of the operational scene are acquired through a camera device. This camera device refers to a device capable of recording video of the current operational scene in real time. The operational scene includes container lifting and / or container return operations. Container lifting refers to the process of a spreader lifting a container from a truck and placing it onto a cargo ship. Container return refers to the process of a spreader lifting a container from a cargo ship and placing it onto a truck. A video stream is a continuous sequence of images, essentially composed of a set of consecutive image frames. Optionally, the video stream may include video of the entire operational process from the start to the end of the operation.

[0132] In this embodiment, the camera device can be installed outside the quay crane to acquire video streams of the truck, spreader, and container within the operational scene. In other embodiments, the camera device can also be installed on the quay crane, as long as it can record the entire process of spreader operation; the specific installation location is not limited.

[0133] It should be noted that the number of camera devices can be set according to actual needs. Specifically, it can be set according to the shooting angle and shooting range of the camera devices, or according to factors such as cost. For example, there can be one, two, or three camera devices, etc., and there is no limitation here.

[0134] Specifically, after acquiring the video stream of the work scene, the video stream needs to be decoded to obtain multiple image frames. Each image frame is the smallest visual unit that makes up the video stream; it is a static image. Combining a temporally consecutive sequence of image frames forms a dynamic video stream.

[0135] For example, to facilitate subsequent detection and recognition, the video stream can be decoded using technologies such as FFmpeg to obtain multiple image frames. FFmpeg is an open-source computer program that can record, convert, and stream digital audio and video. Licensed under the LGPL or GPL, it provides a complete solution for recording, converting, and streaming audio and video, and includes the advanced audio / video codec library libavcodec. To ensure high portability and encoding / decoding quality, much of the code in libavcodec can be developed from scratch, facilitating data use and adaptation. Accordingly, in practical applications, video streams from captured work scenes can also be saved and transmitted using FFmpeg technology.

[0136] S102, based on multiple image frames, determines the number of target pauses and the duration of target pauses during the operation of the spreader.

[0137] Specifically, after obtaining multiple image frames, the target number of pauses and the target pause duration during the operation can be determined based on these image frames. The target number of pauses refers to the total number of pauses the spreader makes during the operation. The target pause duration refers to the total pause duration the spreader makes during the operation.

[0138] S103, based on the target number of pauses and the target pause duration, determine whether the operator's operating behavior is standardized. The operator is the person who controls the lifting device to carry out the operation.

[0139] Specifically, after obtaining the target number of pauses and the target pause duration during the operation, the operator's behavior can be determined as standardized based on these figures. The operator refers to the person responsible for controlling the spreader during the operation. In this embodiment, the operator controls the spreader from the quay crane's cab.

[0140] In one possible implementation, if the target number of pauses of the spreader is greater than or equal to a preset number, and the target pause duration of the spreader exceeds a preset duration, the operator's operational behavior guidelines are determined. It is understood that the preset number of pauses can be set according to the actual situation of the work cycle. For example, if the work cycle only includes the spreader grabbing a container from the truck, or the spreader placing a container onto the truck, then the preset number of pauses is 1; if the work cycle includes the process of the spreader lifting a container from the cargo ship and placing it onto an inner truck at the port terminal, or the process of the spreader lifting a container from the truck and placing it onto the cargo ship, then the preset number of pauses is 2.

[0141] For example, the preset time can be set according to actual needs. For instance, the preset time can be 3 seconds or 2 seconds, etc., without any specific limitation.

[0142] In another possible implementation, if the target number of pauses of the spreader is less than the preset number, or if the target pause duration of the spreader does not exceed the preset duration, the operator's operation behavior is determined to be non-standard.

[0143] In this embodiment, by determining whether the target number of pauses of the spreader is less than the preset number and whether the target pause duration of the spreader exceeds the preset duration, it is possible to determine whether the operator's behavior is standardized. This can improve the accuracy of determining the behavior standard, thereby reducing the occurrence of container falls and improving the safety of spreader operations.

[0144] In this embodiment of the application, by determining the target number of pauses and the target pause duration during the operation of the spreader, it is possible to determine whether the operator's behavior is standardized. This can improve the accuracy of determining the behavior standard, effectively reduce the occurrence of container falls, prevent container fall incidents from the root, and improve the safety of spreader operations.

[0145] In one possible implementation, if the operator's behavior is in accordance with regulations, it indicates that the current operation is compliant, and therefore, monitoring of the next operation can begin.

[0146] Optionally, if the operator's actions are not in accordance with regulations, a prompt message can be output. This prompt message includes, but is not limited to, voice prompts, sound prompts (such as alarm sounds), graphic prompts, and light prompts. In this embodiment, an audible and visual alarm installed in the quay crane operator's cab outputs sound and light alarms. In other embodiments, the graphic and text prompt message can be displayed on a screen in the quay crane operator's cab, or a voice broadcast device in the quay crane operator's cab can output voice prompts or alarm sounds. Furthermore, the prompt message can also be sent to the operator's handheld terminal (such as a mobile phone); the specific method of outputting the prompt message is not limited. This allows the operator to clearly understand whether their actions are in accordance with regulations, facilitating timely adjustments.

[0147] Optionally, if the operator's actions are not standardized, facial recognition is performed on the operator to obtain their identity information, and this information is then reported. Facial recognition is a biometric technology that identifies individuals based on their facial features. It uses a camera or webcam to capture images or video streams containing faces. First, it determines whether a face exists. If a face is present, it further provides the location, size, and position information of each major facial feature. Based on this information, it extracts the identity features contained in each face and compares them with known faces to identify the identity of each face.

[0148] In this embodiment, the operator's facial information can be acquired in real time by a camera installed in the quay crane operator's cab. The operator's identity information is then obtained by recognizing their face using a facial recognition algorithm. The facial recognition algorithm can be the Seetface6 open-source face detection and recognition algorithm.

[0149] For example, a penalty notice containing the violation incident and the identity information of the violator can be automatically generated based on the operator's identity information, and the penalty notice can be reported to the port management personnel's mobile terminal (such as a computer).

[0150] In one possible implementation, see S102 above. Figure 2 The diagram shown is a flowchart of a method for determining the target number of pauses and the target pause duration according to an embodiment of this application, including S1021 to S1024:

[0151] S1021, for each image frame, target detection is performed on the image frame based on the first target neural network to obtain the detection result of the image frame. The detection result is used to indicate the position information of the spreader and / or the truck.

[0152] Specifically, an image frame can be input into a first target neural network for target detection to obtain the detection result of the image frame. In an image frame, a closed region distinct from the surrounding environment is often referred to as the target. The process of determining the location of the target in the image is called detection. In this embodiment, the first target neural network can be a YOLOv5 single-stage target detection model. In other embodiments, the first target neural network can also be a deep residual network, etc., and there is no specific limitation.

[0153] In one possible implementation, the training process of the first target neural network is as follows: acquiring an image sample set, the image sample set including multiple image frames obtained by decoding a video actually captured by a camera device, the multiple image frames having category labels, the category labels including at least one such as empty lifting equipment, lifting equipment with a box, and container truck; based on the image sample set and the category label of each image sample, performing supervised training on the neural network to be trained to obtain the trained first target neural network.

[0154] Specifically, image samples can be used as input to a neural network to be trained. The neural network learns and predicts the image samples, outputting the predicted category label corresponding to the image sample. Based on the predicted category label corresponding to the image sample and the category label labeled on the image sample, the parameters of the neural network to be trained are adjusted until the first target neural network is trained.

[0155] The detection results of the image frames include detection boxes for the lifting equipment and / or the container trucks. The detection boxes for the lifting equipment indicate its position information; for example, the lifting equipment in the image frame can be marked with a detection box, and its position information can be represented by the coordinates of the four corners of the detection box. Similarly, the detection boxes for the container trucks indicate their position information; for example, the container trucks in the image frame can be marked with a detection box, and their position information can be represented by the coordinates of the four corners of the detection box.

[0156] S1022, based on the detection results of the first group of image frames in multiple image frames, determine whether the spreader has entered the working state. The first group of image frames includes the first target image frame and all image frames before the first target image frame.

[0157] Specifically, after obtaining the detection results of multiple image frames, it can be determined whether the lifting device has entered the working state based on the detection results of the first group of image frames among the multiple image frames. The first group of image frames includes the first target image frame and all image frames preceding the first target image frame. The first target image frame is any image frame among the multiple image frames.

[0158] In one possible implementation, if the first set of image frames includes both the truck and the spreader, and the truck's position information remains unchanged, and the detection result of the first target image frame indicates that the spreader is directly above the truck, then the spreader is determined to be in operation. It should be noted that the spreader can be an empty spreader or a spreader with a container; there is no specific limitation.

[0159] In another possible implementation, if any image frame in the first set of image frames does not contain the truck or the lifting device, or if the position information of the truck changes, or if the detection result of the first target image frame indicates that the lifting device is not directly above the truck, it is determined that the lifting device has not entered the working state.

[0160] In this embodiment, the presence or absence of a container truck and a lifting device is determined by checking whether the first set of image frames includes both the container truck and the lifting device, whether the container truck's position information has changed, and whether the lifting device is located directly above the container truck. This method provides a basis for accurate determination of whether the lifting device has entered the working state, thus providing a foundation for subsequent inspection work.

[0161] S1023, when the spreader enters the working state, based on the detection results of the second group of image frames in multiple image frames, determine whether the current operation has ended. The second group of image frames includes the second target image frame and all image frames before the second target image frame. The second target image frame is the image frame after the first target image frame.

[0162] Specifically, after determining that the spreader has entered the operational state, the current operation can be determined to be complete based on the detection results of the second group of image frames among multiple image frames. The second group of image frames includes the second target image frame and all image frames preceding it. The second target image frame is any image frame following the first target image frame.

[0163] In one possible implementation, see Figure 3 The diagram shown is a flowchart of a method for determining whether a current job has ended, provided in an embodiment of this application, including the following steps S301 to S303:

[0164] S301, based on the detection results of the second set of image frames, determine the motion state of the spreader and the motion state of the truck.

[0165] Specifically, after determining that the spreader has entered the working state, the motion state of the spreader and the motion state of the truck can be determined based on the detection results of the second set of image frames.

[0166] In one possible implementation, see Figure 4 The diagram shown is a flowchart of a method for determining the motion state of a lifting device and a container truck according to an embodiment of this application, including the following steps S401 to S402:

[0167] S401, based on the position information of the truck indicated by the detection results of the second set of image frames, determine the motion state of the truck, which includes left and right movement or stationary state.

[0168] Specifically, after determining that the spreader has entered the working state, the movement state of the container truck can be determined based on the position information of the truck indicated by the detection results of the second set of image frames. The movement state of the container truck includes left and right movement or a stationary state.

[0169] It is understandable that the movement state of the truck is determined based on changes in its position information. For example, taking the second set of image frames as an example, where image frame A precedes image frame B, if the truck in image frame A is to the left of the truck in image frame B, then the movement state of the truck can be determined to be left-right movement.

[0170] S402, based on the detection results of the second target neural network and the second target image frame, indicate the position information of the lifting device and determine the motion state of the lifting device. The motion state of the lifting device includes a stationary state, upward movement, downward movement, or left and right movement.

[0171] Specifically, after determining that the spreader has entered the working state, the motion state of the spreader can be determined based on the detection results of the second target neural network and the second set of image frames. The motion state of the spreader includes a stationary state, upward movement, downward movement, or left-right movement.

[0172] In this embodiment, since the movement range of the lifting device is relatively small, if the first target neural network is used for target detection, the detection results may be inaccurate. Therefore, in order to improve the accuracy of detection, this application uses a second neural network for target detection, such as the GCNv2 feature extraction network.

[0173] In one possible implementation, see Figure 5 The diagram shown is a flowchart of a method for determining the motion state of a lifting device according to an embodiment of this disclosure, including the following steps S501 to S505:

[0174] S501, based on the position information of the lifting device indicated by the detection result of the second target image frame, determine the maximum working area of ​​the lifting device.

[0175] Specifically, the maximum working area B1(x,y,w,H) of the spreader can be determined based on the coordinates of the four corners of the detection box of the spreader contained in the detection result of the second target image frame. For example, detection box B0(x,y,w,h), where (x,y) are the coordinates of the top left corner of the detection box, w is the width of the detection box, and h is the height of the detection box. Here, (x,y,w) are the same as the values ​​in detection box B0, and H is the height of the second target image frame. This can improve the accuracy of determining the motion state of the spreader.

[0176] S502, based on the second target neural network, features are extracted from the maximum working area of ​​the lifting device to obtain the feature points of the lifting device in the second target image frame.

[0177] Specifically, the maximum working area of ​​the lifting device can be input into the second target neural network for feature extraction to obtain the feature points of the lifting device in the second target image frame.

[0178] S503, acquire the feature points of the lifting device in all image frames except the second target image frame in the second group of image frames.

[0179] Specifically, after obtaining the feature points of the lifting device in the second target image frame, the feature points of the lifting device in all image frames preceding the second target image frame can be obtained. It can be understood that the feature points of the lifting device in all image frames are obtained by determining the maximum working area of ​​the lifting device based on the detection box of the lifting device in the image frame, and then inputting the maximum working area of ​​the lifting device into the second target neural network for feature extraction; this will not be elaborated further here.

[0180] S504, based on the feature points of the lifting device in the second set of image frames, determine the target line segment of all feature points within the maximum working area.

[0181] Specifically, after acquiring the feature points of the spreader in all image frames of the second set of image frames except for the second target image frame, the target line segment within the maximum working area can be determined based on the feature points of the spreader in the second set of image frames. Here, the target line segment refers to the line segment connecting all feature points within the maximum working area. It can be understood that the movement trajectory of the spreader in the second set of image frames can be determined based on this target line segment.

[0182] S505, based on the target line segment, determines the motion state of the lifting device.

[0183] Specifically, after obtaining the target line segment of all feature points within the maximum working area, it is necessary to analyze and vote on the target line segment to obtain the motion state of the lifting device.

[0184] For example, if there are 10 feature points within the maximum working area, and 8 feature points indicate that the spreader's movement is upward, while 2 feature points indicate that the spreader's movement is downward, then the two feature points indicating downward movement can be ignored, and the spreader's movement can be determined as upward movement. This improves the accuracy of movement state determination, reduces the occurrence of errors in determining the spreader's lifting state, and provides a basis for subsequently determining whether the current operation has ended.

[0185] S302, based on the motion state of the spreader and the motion state of the truck, determine whether the spreader is in the lifting state.

[0186] Specifically, after obtaining the motion status of the spreader and the truck, it can be determined whether the spreader is in a lifting state based on their motion status.

[0187] For example, if the truck is stationary and the spreader is moving upwards, the spreader is determined to be in a lifting state. If the truck is moving left or right, or the spreader is moving in any other state besides upwards, the spreader is determined to be in a non-lifting state. Other states include stationary, downwards, or left and right movements. This allows for precise determination of whether the spreader is in a lifting state, thereby improving the accuracy of determining whether the current operation has ended.

[0188] S303, when the spreader is in the lifting state, based on the detection result of the third group of image frames in the second group of image frames, determine whether the current operation has ended. The third group of image frames is the last X image frames in the second group of image frames, where X is a preset value.

[0189] Specifically, after confirming that the lifting device is in the lifting state, the current operation can be determined to be complete based on the detection results of the third set of image frames in the second set of image frames. The third set of image frames refers to the last X image frames in the second set of image frames, where X is a preset value. The preset value can be set according to actual needs; for example, it could be 5, 7, etc., without any specific limitation.

[0190] In one possible implementation, the current operation is determined to be terminated if the detection result of the second target image frame indicates that the second target image frame does not include the truck, or if the detection result of the third set of image frames indicates that the third set of image frames does not include the lifting equipment.

[0191] In another possible implementation, if the detection result of the second target image frame indicates that the second target image frame includes a truck, and / or the detection result of the third set of image frames indicates that any image frame in the third set of image frames includes a spreader, then it is determined that the current operation has not ended. In this way, it is possible to accurately determine whether the current operation has ended, thereby making the number of target pauses and the duration of target pauses more precise, reducing the occurrence of container falls.

[0192] S1024, if it is determined that the current job has ended, determine the number of target pauses and the target pause duration based on the detection results of multiple image frames.

[0193] Specifically, after the current operation is completed, the number of target pauses and the duration of each pause can be determined based on the detection results of multiple image frames. This improves the accuracy of determining the number of target pauses and the duration of each pause, further enhancing the accuracy of determining the behavioral norms and effectively reducing the probability of container falls.

[0194] In one possible implementation, after determining that the current operation has not yet ended, the first number of pauses and the first pause duration of the spreader can be determined based on the movement state of the spreader in the second set of image frames. A third target image frame is then determined from multiple image frames, and this third target image frame is the next image frame after the second target image frame. The second set of image frames and the third target image frame are then redefined as a new second set of image frames, and based on this new second set of image frames, it is determined whether the current operation has ended. This ensures that the first number of pauses and the first pause duration obtained at the end of the current operation are accurate, improving the accuracy of the behavior specification determination, thereby reducing the occurrence of container falls and improving the safety of spreader operations.

[0195] Specifically, if the motion state of the lifting device changes from upward movement to a stationary state in the second set of image frames, the first pause count of the lifting device is increased by 1; if the motion state of the lifting device remains stationary in the second set of image frames, the first pause duration of the lifting device is increased by the target duration. The target duration is determined by the frame rate of the video stream. The frame rate is the frequency (rate) at which bitmap images, in units of frames, appear continuously on the display. For example, if the frame rate includes 24 image frames per second, the target duration is 1 / 24 of a second.

[0196] For example, if the second set of image frames includes 5 image frames, and the lifting device moves upward in the first 4 image frames and remains stationary in the 5th image frame, then the number of pauses for the lifting device can be increased by 1. If the video to be detected includes 5 image frames, and the lifting device moves upward in the first 3 image frames and remains stationary in the 4th and 5th image frames, then the pause time for the lifting device can be increased to 2 target durations. This improves the accuracy of the first pause duration and its determination, providing a basis for subsequently determining the target pause duration and its target duration.

[0197] It can be understood that the target number of pauses during the operation of the spreader is the first pause corresponding to the end of the current operation, and the target pause duration during the operation is the first pause duration corresponding to the end of the current operation. In this way, the target pause duration and target pause duration generated at the end of the current operation are determined based on the first pause duration and target pause duration, improving the accuracy of the determination of the target pause duration and target pause duration, thereby improving the accuracy of the determination of the behavioral norms.

[0198] It should be noted that the current implementation method involves judging and detecting the video stream generated after the current operation ends, to determine whether the lifting equipment has entered the operation state and whether the current operation has ended. However, in actual implementation, this application acquires the video of the scene area to be detected in real time, and then determines whether the lifting equipment has entered the operation state based on the detection results of multiple image frames in the video. After determining that the lifting equipment has entered the operation state, it continues to judge whether the current operation has ended. After determining that the current operation has ended, it determines the target number of pauses and the target pause duration. Finally, based on the target number of pauses and the target pause duration, it determines whether the operator's operation behavior is standardized.

[0199] It is understandable that the image frame corresponding to when the lifting device enters the working state will not be the image frame corresponding to when the current work ends. Therefore, it is necessary to reacquire the video to be detected in the scene area and detect the video to be detected. However, when detecting, it is not necessary to determine whether the lifting device has entered the working state again. It is only necessary to determine whether the current work has ended based on the detection results of the image frames in the video to be detected.

[0200] In one possible implementation, see Figure 6 The diagram shows a structural schematic of a system for detecting operational behavior norms provided in this application. Specifically, the method for determining operational behavior norms in this application runs on an artificial intelligence (AI) server. A camera device 1, responsible for capturing the quay crane's operation process, is installed outside the quay crane, while a camera device 2, responsible for capturing the faces of operators, is installed in the quay crane's driver's cab. Camera devices 1 and 2 can transmit video streams of the operational scene and facial video streams from the quay crane's driver's cab to the AI ​​server in real time via 5G-CPE (5G Customer Premise Equipment) and 5G base stations. The method for determining operational behavior norms running on the AI ​​server identifies the behavior in real time and reports the results to a safety management terminal (such as a computer). By monitoring the operational behavior norms of operators in real time, the risk of containers falling from the quay crane is reduced at the source, improving the efficiency and level of port quay crane operation safety management. Furthermore, the AI ​​server can also control the audible and visual alarm 3 to issue an alarm based on the identification results, reminding operators to correct their operational behavior promptly.

[0201] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0202] Based on the same inventive concept, this application also provides an operating behavior specification determination device corresponding to the operating behavior specification determination method. Since the principle of the device in this application to solve the problem is similar to the operating behavior specification determination method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0203] Reference Figure 7 The diagram shown is a structural schematic of an operational behavior specification determination device 700 provided in an embodiment of this application. The operational behavior specification determination device 700 includes:

[0204] The video acquisition module 701 is used to acquire the video stream of the work scene and decode the video stream to obtain multiple image frames. The work scene includes lifting the container with a spreader and / or returning the container with a spreader.

[0205] The first determining module 702 is used to determine the number of target pauses and the duration of target pauses of the spreader during the operation based on multiple image frames;

[0206] The second determining module 703 is used to determine whether the operator's operation behavior is standardized based on the target number of pauses and the target pause duration. The operator is the person who controls the lifting device to carry out the operation.

[0207] In one possible implementation, the first determining module 702 is specifically used for:

[0208] For each image frame, target detection is performed on the image frame based on the first target neural network to obtain the detection result of the image frame. The detection result is used to indicate the position information of the spreader and / or the truck.

[0209] Based on the detection results of the first group of image frames in multiple image frames, it is determined whether the spreader has entered the working state. The first group of image frames includes the first target image frame and all image frames before the first target image frame.

[0210] When the spreader enters the working state, based on the detection results of the second group of image frames in multiple image frames, it is determined whether the current operation has ended. The second group of image frames includes the second target image frame and all image frames before the second target image frame. The second target image frame is the image frame after the first target image frame.

[0211] If the current task is determined to be completed, the number of target pauses and the duration of target pauses are determined based on the detection results of multiple image frames.

[0212] In one possible implementation, the first determining module 702 is specifically used for:

[0213] If the first set of image frames includes both the truck and the spreader, and the position information of the truck has not changed, and the detection result of the first target image frame indicates that the spreader is located directly above the truck, then the spreader is determined to have entered the working state.

[0214] In one possible implementation, the first determining module 702 is specifically used for:

[0215] Based on the detection results of the second set of image frames, the motion state of the spreader and the motion state of the truck are determined.

[0216] Determine whether the spreader is in a lifting state based on the movement state of the spreader and the truck.

[0217] When the lifting device is in the lifting state, the detection results of the third set of image frames in the second set of image frames determine whether the current operation has ended. The third set of image frames is the last X image frames in the second set of image frames, where X is a preset value.

[0218] In one possible implementation, the first determining module 702 is specifically used for:

[0219] Based on the position information of the truck indicated by the detection results of the second set of image frames, the motion state of the truck is determined, including left and right movement or stationary state.

[0220] Based on the detection results of the second target neural network and the second target image frame, the position information of the lifting device is indicated, and the motion state of the lifting device is determined. The motion state of the lifting device includes a stationary state, upward movement, downward movement, or left and right movement.

[0221] In one possible implementation, the first determining module 702 is specifically used for:

[0222] Based on the position information of the lifting device indicated by the detection results of the second target image frame, the maximum working area of ​​the lifting device is determined;

[0223] Based on the second target neural network, feature extraction is performed on the maximum working area of ​​the lifting device to obtain the feature points of the lifting device in the second target image frame;

[0224] Obtain the feature points of the lifting device in all image frames except the second target image frame in the second group of image frames;

[0225] Based on the feature points of the lifting device in the second set of image frames, the target line segments of all feature points within the maximum working area are determined.

[0226] Based on the target line segment, determine the motion state of the lifting device.

[0227] In one possible implementation, the first determining module 702 is specifically used for:

[0228] When the truck is stationary and the spreader is moving upwards, the spreader is determined to be in a lifting state.

[0229] In one possible implementation, the first determining module 702 is specifically used for:

[0230] If the detection result of the second target image frame indicates that the second target image frame does not include the truck, or if the detection result of the third set of image frames indicates that the third set of image frames does not include the lifting equipment, the current operation is determined to be terminated.

[0231] In one possible implementation, the first determining module 702 is specifically used for:

[0232] If the current operation has not been completed, the first number of pauses of the spreader and the duration of the first pause are determined based on the motion state of the spreader in the second set of image frames.

[0233] The third target image frame is determined from multiple image frames, and the third target image frame is the next image frame after the second target image frame;

[0234] The second set of image frames and the third target image frame are redefined as a new second set of image frames, and based on the new second set of image frames, it is determined whether the current operation has ended.

[0235] In one possible implementation, the first determining module 702 is specifically used for:

[0236] If the motion state of the lifting device changes from upward movement to a stationary state in the second set of image frames, the first pause count of the lifting device will be increased by 1.

[0237] If the motion state of the lifting device remains stationary in the second set of image frames, the first pause duration of the lifting device is increased by a target duration, which is determined by the frame rate of the video stream.

[0238] In one possible implementation, the target number of pauses is the first number of pauses corresponding to the end of the current task, and the target pause duration is the first pause duration corresponding to the end of the current task.

[0239] In one possible implementation, the second determining module 703 is specifically used for:

[0240] If the target number of pauses in the spreader is greater than or equal to the preset number, and the target pause duration exceeds the preset duration, then the operator's operational behavior guidelines shall be determined; or,

[0241] If the number of target pauses of the spreader is less than the preset number, or if the target pause duration of the spreader does not exceed the preset duration, the operator's operation is determined to be non-standard.

[0242] See Figure 8 As shown, in one possible implementation, the device 700 for determining operational behavior specifications further includes:

[0243] Information output module 704 is used to output a prompt message when it is determined that the operator's operation is not in accordance with regulations. The prompt message is used to remind the operator; and / or,

[0244] If it is determined that the operator's operation is not in accordance with regulations, facial recognition is performed on the operator to obtain the operator's identity information, and the operator's identity information is reported.

[0245] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0246] Based on the same technical concept, embodiments of this application also provide an electronic device. (Refer to...) Figure 9 The diagram shown is a structural schematic of an electronic device 900 provided in an embodiment of this application, including a processor 901, a memory 902, and a bus 903. The memory 902 is used to store execution instructions and includes a main memory 9021 and an external memory 9022. The main memory 9021, also called internal memory, is used to temporarily store computational data in the processor 901, as well as data exchanged with external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the main memory 9021.

[0247] In this embodiment, the memory 902 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 901. That is, when the electronic device 900 is running, the processor 901 communicates with the memory 902 through the bus 903, so that the processor 901 executes the application code stored in the memory 902, and thus executes the method disclosed in any of the foregoing embodiments.

[0248] The memory 902 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0249] Processor 901 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0250] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 900. In other embodiments of this application, the electronic device 900 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0251] This application also provides a computer-readable storage medium storing a computer program. When a processor runs the computer program, it executes the method for determining the operational behavior specifications described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0252] This application also provides a computer program product that carries program code. The instructions included in the program code can be used to execute the method for determining the operational behavior specifications described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0253] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0254] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0255] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0256] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0257] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0258] It should be understood that if the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the user of the personal information processing rules and obtained the user's consent before processing the personal information.

[0259] If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the requirement of "express consent".

[0260] For example, at personal information collection devices such as cameras, clear and prominent signs should be set up to inform individuals that they have entered the scope of personal information collection and that their personal information will be collected. If an individual voluntarily enters the collection scope, it is considered that they have agreed to the collection of their personal information. Alternatively, on personal information processing devices, with clear signs / information informing individuals of the personal information processing rules, authorization can be obtained through pop-up messages or by asking individuals to upload their personal information themselves. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0261] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art, within the scope of the technology disclosed in this disclosure, can apply its knowledge to other applications.

[0262] However, modifications or easily conceivable variations can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications, variations, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for determining operational behavior norms, characterized in that, include: Acquire a video stream of the work scene and decode the video stream to obtain multiple image frames. The work scene includes lifting and / or returning of the lifting equipment. For each image frame, target detection is performed on the image frame based on the first target neural network to obtain the detection result of the image frame. The detection result is used to indicate the position information of the spreader and / or the container truck. Based on the detection results of the first group of image frames among the plurality of image frames, it is determined whether the lifting device has entered the working state. The first group of image frames includes the first target image frame and all image frames before the first target image frame. When the spreader enters the working state, based on the detection results of the second group of image frames in the plurality of image frames, it is determined whether the current operation has ended. The second group of image frames includes the second target image frame and all image frames before the second target image frame. The second target image frame is the image frame after the first target image frame. If the current operation is determined to be completed, the number of target pauses and the duration of target pauses during the operation are determined based on the detection results of the multiple image frames. Based on the target number of pauses and the target pause duration, it is determined whether the operator's operating behavior is standardized. The operator is the person who controls the lifting device to perform the operation.

2. The method according to claim 1, characterized in that, Determining whether the lifting device has entered the working state based on the detection result of the first group of image frames among the plurality of image frames includes: If the first set of image frames includes both the container truck and the lifting device, and the position information of the container truck has not changed, and the detection result of the first target image frame indicates that the lifting device is located directly above the container truck, then it is determined that the lifting device has entered the working state.

3. The method according to claim 1, characterized in that, Determining whether the current task has ended based on the detection results of the second group of image frames among the plurality of image frames includes: Based on the detection results of the second set of image frames, the motion state of the lifting device and the motion state of the container truck are determined; Based on the motion state of the spreader and the motion state of the truck, determine whether the spreader is in a lifting state; When the lifting device is in the lifting state, based on the detection result of the third group of image frames in the second group of image frames, it is determined whether the current operation has ended. The third group of image frames is the last X image frames in the second group of image frames, where X is a preset value.

4. The method according to claim 3, characterized in that, The determination of the motion state of the lifting device and the motion state of the container truck based on the detection results of the second set of image frames includes: Based on the position information of the truck indicated by the detection results of the second set of image frames, the motion state of the truck is determined, and the motion state of the truck includes left and right movement or a stationary state; Based on the detection results of the second target neural network and the second target image frame, the position information of the lifting device is indicated, and the motion state of the lifting device is determined. The motion state of the lifting device includes a stationary state, upward movement, downward movement, or left and right movement.

5. The method according to claim 4, characterized in that, The determination of the motion state of the lifting device based on the position information of the lifting device indicated by the detection results of the second target neural network and the second target image frame includes: Based on the position information of the lifting device indicated by the detection result of the second target image frame, the maximum working area of ​​the lifting device is determined; Based on the second target neural network, feature extraction is performed on the maximum working area of ​​the lifting device to obtain the feature points of the lifting device in the second target image frame; Obtain the feature points of the lifting device in all image frames of the second group of image frames, excluding the second target image frame; Based on the feature points of the spreader in the second set of image frames, the target line segments of all feature points within the maximum working area are determined. Based on the target line segment, the motion state of the lifting device is determined.

6. The method according to claim 4, characterized in that, Determining whether the spreader is in a lifting state based on the motion state of the spreader and the motion state of the truck includes: When the truck is stationary and the spreader is moving upwards, the spreader is determined to be in a lifting state.

7. The method according to claim 3, characterized in that, Determining whether the current task has ended based on the detection results of the third group of image frames in the second group of image frames includes: If the detection result of the second target image frame indicates that the truck is not included in the second target image frame, or if the detection result of the third group of image frames indicates that the lifting device is not included in any of the third group of image frames, then the current operation is determined to be terminated.

8. The method according to claim 1, characterized in that, The method further includes: If the current operation has not ended, the first number of pauses of the spreader and the first pause duration of the spreader are determined based on the motion state of the spreader in the second set of image frames; A third target image frame is determined from the plurality of image frames, wherein the third target image frame is the next image frame after the second target image frame; The second group of image frames and the third target image frame are redefined as a new second group of image frames, and based on the new second group of image frames, it is determined whether the current operation has ended.

9. The method according to claim 8, characterized in that, The step of determining the number of pauses and the duration of each pause of the lifting device based on the motion state of the lifting device in the second set of image frames includes: If the motion state of the lifting device changes from upward movement to a stationary state in the second set of image frames, the first pause count of the lifting device is increased by 1; If the motion state of the lifting device remains stationary in the second set of image frames, the first pause duration of the lifting device is increased by a target duration, the target duration being determined by the frame rate of the video stream.

10. The method according to claim 9, characterized in that, The target number of pauses is the first number of pauses corresponding to the end of the current task, and the target pause duration is the first pause duration corresponding to the end of the current task.

11. The method according to claim 10, characterized in that, The determination of whether the operator's behavior is standardized based on the target number of pauses and the target pause duration includes: If the target number of pauses of the spreader is greater than or equal to a preset number, and the target pause duration of the spreader exceeds a preset duration, then the operator's operational behavior guidelines are determined; or, If the target number of pauses of the lifting device is less than the preset number, or if the target pause duration of the lifting device does not exceed the preset duration, the operator's operation behavior is determined to be non-standard.

12. The method according to claim 1 or 11, characterized in that, The method further includes: If it is determined that the operator's operation is not in accordance with regulations, a prompt message is output to remind the operator; and / or, If it is determined that the operator's operation is not in accordance with regulations, facial recognition is performed on the operator to obtain the operator's identity information, and the operator's identity information is reported.

13. A device for determining operational behavior norms, characterized in that, include: The video acquisition module is used to acquire the video stream of the operation scene and decode the video stream to obtain multiple image frames. The operation scene includes lifting the container with a spreader and / or returning the container with a spreader. The first determining module is used to perform target detection on each image frame based on a first target neural network to obtain the detection result of the image frame, and the detection result is used to indicate the position information of the lifting device and / or the container truck. The first determining module is further configured to determine whether the lifting device has entered the working state based on the detection result of the first group of image frames in the plurality of image frames, wherein the first group of image frames includes the first target image frame and all image frames preceding the first target image frame; The first determining module is further configured to determine whether the current operation has ended based on the detection result of the second group of image frames in the plurality of image frames when the spreader enters the operation state. The second group of image frames includes the second target image frame and all image frames before the second target image frame. The second target image frame is the image frame after the first target image frame. The first determining module is further configured to, upon determining that the current operation has ended, determine the number of target pauses and the duration of target pauses of the lifting device during the operation based on the detection results of the multiple image frames; The second determining module is used to determine whether the operator's operation behavior is standardized based on the target number of pauses and the target pause duration, wherein the operator is the operator who controls the lifting device to perform the operation.

14. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the method for determining the operating behavior specifications as described in any one of claims 1 to 12 is performed.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method for determining operational behavior specifications as described in any one of claims 1 to 12.

Images