Loading state recognition method, device and equipment and computer readable storage medium
By generating image frame sequences of forklifts, and using behavior recognition models and state machines to correct the forklift loading status, the problem of forklift loading status recognition being easily interfered with is solved, the recognition accuracy is improved, and the accuracy of warehouse management is ensured.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SF TECH CO LTD
- Filing Date
- 2021-09-06
- Publication Date
- 2026-06-16
Smart Images
Figure CN115775361B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, specifically to a loading status recognition method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] In warehousing scenarios, forklifts are typically used for loading and unloading goods. Therefore, monitoring video information from the warehousing environment can be used to monitor the loading status of forklifts, thereby determining their working efficiency and achieving automated warehouse management.
[0003] In existing technologies, the loading status of forklifts is usually determined based on image recognition. However, when there is interference in the image or the acquired image is faulty, incorrect loading status recognition results are often obtained, affecting the effectiveness of subsequent automated management.
[0004] It is evident that existing forklift loading status recognition methods suffer from technical problems, such as susceptibility to image interference, leading to insufficient recognition accuracy. Summary of the Invention
[0005] This application provides a loading status recognition method, apparatus, device, and computer-readable storage medium, aiming to solve the technical problems of existing forklift loading status recognition methods being susceptible to image interference and having insufficient recognition accuracy.
[0006] On one hand, embodiments of this application provide a loading status identification method, including:
[0007] Obtain the video stream corresponding to the target forklift;
[0008] Generate multiple image frame sequences based on the video stream;
[0009] For each image frame sequence, the forklift loading action information corresponding to the image frame sequence is determined based on the multiple images contained in the image frame sequence.
[0010] The loading status of the target forklift is determined based on the forklift loading action information corresponding to each of the image frame sequences.
[0011] In one embodiment of this application, determining forklift loading action information corresponding to the image frame sequence based on multiple image frames contained in the image frame sequence includes:
[0012] The multiple frames of images contained in the image frame sequence are input into a preset behavior recognition model to obtain the load change feature information of the target forklift;
[0013] Based on the load change feature information, determine the classification confidence level corresponding to each preset target forklift loading action information in the behavior recognition model;
[0014] Set the target forklift loading action information corresponding to the highest classification confidence in the classification confidence as the forklift loading action information corresponding to the image frame sequence.
[0015] In one embodiment of this application, setting the target forklift loading action information corresponding to the maximum classification confidence in the classification confidence scores as the forklift loading action information corresponding to the image frame sequence includes:
[0016] If the maximum classification confidence score in the classification confidence scores is greater than a preset confidence threshold, then the target forklift loading action information corresponding to the maximum classification confidence score in the classification confidence scores is set as the forklift loading action information corresponding to the image frame sequence.
[0017] If the maximum classification confidence score is less than or equal to a preset confidence threshold, then the forklift loading action information corresponding to the previous image frame sequence of the image frame sequence is set as the forklift loading action information corresponding to the image frame sequence.
[0018] In one embodiment of this application, determining the loading state of the target forklift based on forklift loading action information corresponding to each of the image frame sequences includes:
[0019] The forklift loading action information corresponding to each image frame sequence is sorted according to the order of each image frame sequence to obtain the forklift loading action sequence.
[0020] The loading status of the target forklift is determined based on the forklift loading action sequence.
[0021] In one embodiment of this application, determining the loading state of the target forklift based on the forklift loading action sequence includes:
[0022] The forklift loading action sequence is input into a preset state machine to obtain state switching information;
[0023] The previous loading state of the target forklift is switched according to the state switching information to obtain the switched loading state;
[0024] Set the switched loading state to the loading state of the target forklift.
[0025] In one embodiment of this application, generating a plurality of image frame sequences based on the video stream includes:
[0026] The video stream is divided into multiple video stream segments;
[0027] For each video stream segment, the video stream segment is sampled to obtain the image frame sequence of the video stream segment.
[0028] In one embodiment of this application, sampling the video stream segment to obtain an image frame sequence of the video stream segment includes:
[0029] The video stream segment is sampled to obtain multiple sampled images;
[0030] For each sampled image, the sampled image is cropped according to the target detection result of the sampled image to obtain the cropped sampled image;
[0031] Based on the cropped sampled images, an image frame sequence of the video stream segment is generated.
[0032] On the other hand, embodiments of this application also provide a loading status identification device, including:
[0033] The video stream acquisition module is used to acquire the video stream corresponding to the target forklift;
[0034] An image frame sequence generation module is used to generate multiple image frame sequences based on the video stream;
[0035] The action recognition module is used to determine the forklift loading action information corresponding to each image frame sequence based on the multiple images contained in the image frame sequence.
[0036] The status recognition module is used to determine the loading status of the target forklift based on the forklift loading action information corresponding to each of the image frame sequences.
[0037] On the other hand, embodiments of this application also provide a loading status identification device, which includes a processor, a memory, and a loading status identification program stored in the memory and executable on the processor. The processor executes the loading status identification program to implement the steps in the loading status identification method.
[0038] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a loading status identification program, which is executed by a processor to implement the steps in the loading status identification method.
[0039] The loading status recognition method provided in this application first determines the forklift loading action information corresponding to each image frame sequence in a video stream, and then further determines the loading status of the target forklift based on the forklift loading action information corresponding to each image frame sequence. In this way, even if interference or errors in a certain image frame cause the result of the forklift loading action information corresponding to that image frame sequence to be incorrect, it can be corrected based on the result of the forklift loading action information of the remaining image frame sequences, ultimately obtaining an accurate loading status and effectively improving the recognition accuracy of the forklift loading status. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is a schematic diagram of a scenario for the loading status identification method provided in the embodiments of this application;
[0042] Figure 2 This is a flowchart illustrating the first embodiment of the loading status identification method provided in this application.
[0043] Figure 3 This is a flowchart illustrating the second embodiment of the loading status identification method provided in this application.
[0044] Figure 4 This is a flowchart illustrating the third embodiment of the loading status identification method provided in this application.
[0045] Figure 5 This is a flowchart illustrating the fourth embodiment of the loading status identification method provided in this application.
[0046] Figure 6 This is a flowchart illustrating the fifth embodiment of the loading status identification method provided in this application.
[0047] Figure 7 This is a flowchart of the sixth embodiment of the loading status identification method provided in this application.
[0048] Figure 8 This is a flowchart of the seventh embodiment of the loading status identification method provided in this application.
[0049] Figure 9 This is a schematic diagram of an embodiment of the loading status identification device provided in this application.
[0050] Figure 10 This is a schematic diagram of an embodiment of the loading status identification device provided in this application. Detailed Implementation
[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of the present invention.
[0052] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.
[0053] This application provides a loading status identification method, apparatus, device, and computer-readable storage medium, which will be described in detail below.
[0054] The loading status identification method in this application embodiment is applied to a loading status identification device, which is disposed in a loading status identification equipment. The loading status identification equipment includes a memory, a processor, and a loading status identification program stored in the memory and executable on the processor. When the processor executes the loading status identification program, it implements the steps in the loading status identification method.
[0055] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating a scenario in which a loading status recognition method is applied according to an embodiment of this application. The loading status recognition scenario in this embodiment includes a loading status recognition device 100 and a camera 200. The camera 200 is mainly used for monitoring the forklift, specifically for capturing video streams of the forklift in operation and transmitting them to the loading status recognition device 100. The loading status recognition device 100 uses a computer storage medium that runs the loading status recognition method to perform loading status recognition steps on the video stream transmitted from the camera 200, thereby identifying the loading status of the forklift.
[0056] In this embodiment of the invention, the loading status recognition device 100 is mainly used for: acquiring a video stream corresponding to a target forklift; generating multiple image frame sequences based on the video stream; for each image frame sequence, determining forklift loading action information corresponding to the image frame sequence based on the multiple images contained in the image frame sequence; and determining the loading status of the target forklift based on the forklift loading action information corresponding to each image frame sequence.
[0057] It should be noted that, Figure 1 The schematic diagram of the loading status recognition scenario shown is merely an example. The loading status recognition scenario described in this application embodiment is intended to more clearly illustrate the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided in this application embodiment.
[0058] Based on the above-mentioned scenario of loading status recognition, an embodiment of the loading status recognition method is proposed.
[0059] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the first embodiment of the loading status identification method provided in this application. The loading status identification method in this embodiment includes steps 201-204:
[0060] 201. Obtain the video stream corresponding to the target forklift.
[0061] As illustrated in the aforementioned scenario diagram for loading status recognition, the video stream corresponding to the target forklift acquired by the loading status recognition device typically refers to a monitoring video stream containing the target forklift captured by a camera. The monitoring video stream can be transmitted in real-time, allowing for the real-time determination of the target forklift's current loading status, or it can be pre-acquired, allowing for the determination of the target forklift's loading status at various times. For ease of description, this embodiment uses the example of a pre-acquired video stream, where the loading status of the target forklift at any given time is determined through analysis of the video stream. Those skilled in the art will understand that when determining the real-time loading status of a forklift, it is only necessary to ensure that the acquired video stream is a real-time video stream of the target forklift, and the specific steps for processing the video stream remain the same.
[0062] It should be noted that, typically, camera devices monitor the entire scene and not a specific forklift. However, loading status recognition devices can detect targets frame by frame in the monitoring video stream captured by the camera device, thus obtaining a video stream containing the target forklift. In other words, each frame in the video stream corresponding to the target forklift is associated with it.
[0063] 202, Generate multiple image frame sequences based on the video stream.
[0064] In this embodiment, the loading status recognition device can generate multiple image frame sequences of the video stream by segmenting and sampling it. Of course, to avoid losing some motion feature information of the target forklift in the video stream, the sampling frequency of the video stream is limited. Specific generation methods can be found in subsequent sections. Figure 7 And its explanations and descriptions.
[0065] In this embodiment, each image frame sequence contains multiple frames from the video stream. Optionally, each image frame sequence consists of three consecutive frames from the video stream. These three consecutive frames effectively identify the loading action of the target forklift within those three frames. Of course, there are two scenarios where the image frame sequence is derived from three consecutive frames from the video stream. Specifically, taking a video stream that sequentially contains image frames 1, 2, 3, 4, 5, and 6 as an example, the resulting image frame sequence can be (image frame 1, image frame 2, image frame 3) and (image frame 4, image frame 5, image frame 6), or it can be (image frame 1, image frame 2, image frame 3), (image frame 2, image frame 3, image frame 4), (image frame 3, image frame 4, image frame 5) and (image frame 4, image frame 5, image frame 6). The specific method for generating the image frame sequence can be determined based on actual needs, and this application does not impose any limitations on it.
[0066] Of course, if the video stream is a real-time video stream of the target forklift, then the generated image frame sequence can also be understood as a real-time image frame sequence of the target forklift.
[0067] 203. For each image frame sequence, determine the forklift loading action information corresponding to the image frame sequence based on the multiple images contained in the image frame sequence.
[0068] First, it's important to note that, compared to other vehicles, forklifts, due to their inherent characteristics, generally only have two loading states: empty and loaded. That is, when a forklift picks up goods, it is considered to be in a loaded state; when it unloads goods, it is considered to be in an empty state. Correspondingly, there are four types of forklift loading action information: maintaining empty, maintaining loaded, loading, and unloading. Maintaining empty means that multiple frames in the image frame sequence are in an empty state; maintaining loaded means that multiple frames in the image frame sequence are in a loaded state; loading means that multiple frames in the image frame sequence change from an empty state to a loaded state; and unloading means that multiple frames in the image frame sequence change from a loaded state to an empty state. Therefore, the forklift loading action information in each image frame sequence can be determined based on the multiple images contained within the sequence.
[0069] Of course, the forklift loading action information corresponding to the image frame sequence can essentially be considered as the forklift loading action information corresponding to the last frame of the image frame sequence. In other words, it is to use a combination of several consecutive frames in the image frame sequence to determine the forklift loading action information in the last frame of the sequence.
[0070] In this embodiment, there are many ways to implement the loading status recognition device to identify the loading action information of the forklift in the image frame sequence. For example, it can be based on the DT (Dense Trajectories) algorithm and the iDT (improved Dense Trajectories) algorithm. Of course, the most common approach is to directly use a behavior recognition network in artificial intelligence to process multiple frames in the image frame sequence to obtain the loading action information of the forklift in the image frame sequence. Specifically, ECO (Efficient Convolutional Network for Online Video Understanding) can be used as the framework of the behavior recognition network and trained using data samples to obtain an ECO behavior recognition network that can identify the loading action information of the forklift in the image frame sequence. Of course, other behavior recognition networks, such as TSN (Temporal Segment Network) behavior recognition network and TRN (Temporal Relation Network) behavior recognition network, are also feasible, but will not be elaborated on in this invention. The specific implementation process of using the ECO behavior recognition network to identify the loading action information of the forklift in the image frame sequence can be found in the following sections. Figure 3 And its explanations and descriptions.
[0071] Of course, if the image frame sequence is a real-time image frame sequence of the target forklift, then the obtained forklift loading action information is also the real-time loading action information of the forklift.
[0072] 204. Determine the loading status of the target forklift based on the forklift loading action information corresponding to each of the image frame sequences.
[0073] As can be seen from the description in 203 above, the four loading actions included in the forklift loading action information can essentially correspond to the transformation relationship between states. For example, keeping empty and keeping fully loaded can correspond to the transition from empty to empty and from fully loaded to fully loaded states, respectively, while loading and unloading can correspond to the transition from empty to fully loaded and from fully loaded to empty states, respectively.
[0074] It's important to note that since the image frame sequence itself is generated from the video stream, it also possesses a specific sequence. Typically, the loading state of the target forklift in the last image frame sequence is determined by comprehensively analyzing several consecutive image frame sequences; that is, determining the loading state of the target forklift in the last frame of the last image frame sequence. There are many ways to specifically determine the loading state, such as whether the loading action information corresponding to the target forklift in several consecutive image frame sequences meets predetermined state conditions, or whether it meets preset state switching conditions, etc. For specific implementation methods, please refer to the following sections. Figure 5 And its explanations and descriptions.
[0075] In this embodiment, the loading status of the target forklift is comprehensively judged based on the forklift loading action information corresponding to each image frame sequence. This effectively avoids the anomaly of incorrect loading status recognition due to errors in the loading action information of a single forklift. For example, most commonly, when only one image frame sequence shows the forklift loading action as empty, while the remaining image frame sequences show the forklift loading action as fully loaded, the loading status of the target forklift in each of the resulting image frame sequences should be fully loaded, thus avoiding the risk of misjudging the loading status of the target forklift in one of the image frame sequences as empty.
[0076] The loading status recognition method provided in this application first determines the forklift loading action information corresponding to each image frame sequence in a video stream, and then further determines the loading status of the target forklift based on the forklift loading action information corresponding to each image frame sequence. In this way, even if interference or errors in a certain image frame cause the result of the forklift loading action information corresponding to that image frame sequence to be incorrect, it can be corrected based on the result of the forklift loading action information of the remaining image frame sequences, ultimately obtaining an accurate loading status and effectively improving the recognition accuracy of the forklift loading status.
[0077] like Figure 3 As shown, Figure 3 This is a flowchart illustrating the second embodiment of the loading status identification method provided in this application.
[0078] This embodiment proposes a method for processing image frame sequences based on an ECO action recognition network to obtain forklift loading action information corresponding to the image frame sequences. Specifically, it includes steps 301-303:
[0079] 301. Input the multiple frames of images contained in the image frame sequence into a preset behavior recognition model to obtain the load change feature information of the target forklift.
[0080] In this embodiment, the preset behavior recognition model is the ECO action recognition network, which is pre-trained. The ECO action recognition network typically consists of convolutional layers for image feature extraction and classification layers for classifying the extracted features. Considering that the main difference during forklift loading lies in the change in cargo load, the convolutional layers in the trained ECO action recognition network can be used to extract the load feature information of the target forklift in each frame of the image frame sequence, and further obtain the load change feature information of the target forklift in the image frame sequence.
[0081] Specifically, the convolutional layers of the ECO action recognition network mainly include 2D convolutional networks and 3D convolutional networks. The 2D convolutional network is mainly used to process each frame of the image to obtain the load feature information of the target forklift in each frame of the image, while the 3D convolutional network is mainly used to identify the correlation between each frame of the image, that is, to further obtain the load change feature information of the target forklift in the image frame sequence.
[0082] Typically, the load change feature information extracted using the ECO action recognition network is a high-dimensional vector.
[0083] 302. Based on the load change feature information, determine the classification confidence level corresponding to each preset target forklift loading action information in the behavior recognition model.
[0084] In this embodiment, the load change feature information, i.e., the high-dimensional vector, is input into the classification layer of the ECO action recognition network for further processing. This yields the classification confidence scores corresponding to each preset target forklift loading action in the behavior recognition model. The preset target forklift loading action information in the behavior recognition model refers to the four types of forklift loading action information mentioned earlier. Specifically, the classification confidence score corresponding to each target forklift loading action is represented by a normalized four-dimensional vector. Each dimension of the four-dimensional vector corresponds to the classification confidence score of a target forklift loading action, which is also the probability of each target forklift loading action.
[0085] 303, set the target forklift loading action information corresponding to the highest classification confidence in the classification confidence as the forklift loading action information corresponding to the image frame sequence.
[0086] Based on the foregoing description, the classification confidence of forklift loading action information can be understood as the probability of each forklift loading action information corresponding to the image frame sequence. Therefore, the target forklift loading action information corresponding to the highest classification confidence is the most likely forklift loading action information corresponding to the image frame sequence. Thus, the target forklift loading action information can be used as the forklift loading action information corresponding to the image frame sequence.
[0087] As an optional solution, to improve the recognition accuracy of the ECO action recognition network and avoid misjudgments, the loading status recognition device uses a preset confidence threshold to determine whether the forklift loading action information corresponding to a certain image frame sequence is sufficiently reliable. For details, please refer to [link to relevant documentation]. Figure 4 And its explanations and descriptions.
[0088] Furthermore, it should be noted that the ECO action recognition network is trained based on data samples, specifically using sample image frame sequences pre-labeled with forklift loading action information. Since the training process of the ECO action recognition network is similar to other conventional artificial intelligence algorithms, this invention will not elaborate on the specific implementation process of training and generating the ECO action recognition network. The trained ECO action recognition network will be directly stored in the database of the loading status recognition device for real-time retrieval during the loading status recognition process.
[0089] This application embodiment utilizes the ECO action recognition network to extract load change feature information from image frame sequences, and uses the load change feature information to determine the probability of each forklift loading action information. The forklift loading action information with the highest probability is taken as the forklift loading action information corresponding to the image frame sequence, thereby effectively ensuring the accuracy of the forklift loading action information.
[0090] like Figure 4 As shown, Figure 4 This is a flowchart illustrating the third embodiment of the loading status identification method provided in this application.
[0091] In this embodiment, to further improve the recognition accuracy of forklift loading action information, a confidence threshold can be preset. The confidence threshold can be used to determine whether the forklift loading action information corresponding to a certain image frame sequence is sufficiently reliable. Specifically, it includes steps 401 to 403:
[0092] 401. Determine whether the maximum classification confidence score in the classification confidence scores is greater than a preset confidence threshold. If yes, proceed to step 402; otherwise, proceed to step 403.
[0093] In this embodiment, since the classification confidence level describes the probability of forklift loading action information, obviously, the higher the classification confidence level, the more likely the ECO action recognition network considers the image frame sequence to correspond to the forklift loading action information result. Clearly, when the classification confidence levels identified by the ECO action recognition network are relatively balanced, it indicates that the ECO action recognition network also has difficulty effectively identifying the forklift loading action information result corresponding to the image frame sequence. Therefore, by setting a pre-set confidence threshold, when the classification confidence level of forklift loading action information is higher than the confidence threshold, the recognition result of the ECO action recognition network can be considered sufficiently reliable. Conversely, when the classification confidence levels of forklift loading action information are all lower than the confidence threshold, the recognition result of the ECO action recognition network can be considered insufficiently reliable and should be discarded. Therefore, the maximum classification confidence level and the confidence threshold can be compared.
[0094] As an optional approach in this application, when the confidence threshold is set to 0.8, experiments show that this can effectively improve the classification accuracy of forklift loading action information.
[0095] 402, set the target forklift loading action information corresponding to the highest classification confidence in the classification confidence as the forklift loading action information corresponding to the image frame sequence.
[0096] As described above, when the maximum classification confidence score is greater than the confidence threshold, meaning there exists a forklift loading action information whose classification confidence score is higher than the confidence threshold, the recognition result of the ECO action recognition network can be considered sufficiently reliable. Therefore, the loading status recognition device will set the target forklift loading action information corresponding to the maximum classification confidence score as the forklift loading action information corresponding to the image frame sequence.
[0097] 403, set the forklift loading action information corresponding to the previous image frame sequence of the image frame sequence as the forklift loading action information corresponding to the image frame sequence.
[0098] Similarly, as described above, when the maximum classification confidence score is not greater than the confidence threshold (meaning the classification confidence score of each forklift loading action information is not higher than this confidence threshold), the recognition result of the ECO action recognition network can be considered insufficiently reliable. In this case, the loading status recognition device will set the forklift loading action information corresponding to the previous image frame sequence as the forklift loading action information corresponding to this image frame sequence, in order to maximize fault tolerance.
[0099] This embodiment proposes a scheme to verify the recognition results of the ECO action recognition network using a confidence threshold. When it is determined that the recognition results of the ECO action recognition network are not reliable enough, the forklift loading action information corresponding to the target image frame sequence is determined by using the forklift loading action information corresponding to the previous image frame sequence of the target image frame sequence. This maximizes the fault tolerance and further improves the recognition accuracy of the forklift loading action information of the image frame sequence.
[0100] like Figure 5 As shown, Figure 5 This is a flowchart illustrating the fourth embodiment of the loading status identification method provided in this application.
[0101] This embodiment provides a method for obtaining the loading status of a target forklift based on forklift loading action information corresponding to each image frame sequence. Specifically, it includes steps 501-502:
[0102] 501. Sort the forklift loading action information corresponding to each of the image frame sequences according to the order of each of the image frame sequences to obtain the forklift loading action sequence.
[0103] As described in step 204 above, the image frame sequence itself is also a sequence. Therefore, the loading status recognition device sorts the forklift loading action information corresponding to each image frame sequence according to the order of the corresponding image frame sequences. In this way, a sequence describing the changes in forklift loading actions in a continuous image frame sequence can be obtained. Using the obtained forklift loading action sequence can simplify the subsequent process of determining the loading status of the target forklift.
[0104] 502. Determine the loading status of the target forklift based on the forklift loading action sequence.
[0105] In this embodiment, since the forklift loading action sequence contains forklift loading actions from multiple consecutive image frame sequences, the loading status of the target forklift can be accurately determined based on this forklift loading action sequence. Specifically, there are many ways to determine the loading status based on the forklift loading action sequence. Taking a database association as an example, the loading status of the target forklift corresponding to various forklift loading action sequences is pre-stored in the database of the loading status recognition device. For example, most commonly, if the forklift loading action sequence is in the order of keeping empty, loading, and keeping fully loaded, then the loading status of the target forklift corresponding to this sequence in the database is fully loaded.
[0106] As an alternative, besides directly using a relational database to obtain the loading state corresponding to the forklift loading action sequence, the loading state of the target forklift can also be determined using a state machine. A state machine is a mathematical model that can switch between specific states based on input signals. Specifically, since the target forklift's loading state mainly includes two types—fully loaded and empty—the state machine can pre-set the conditions for switching the target forklift's loading state from fully loaded to empty, and vice versa. Thus, when the forklift loading action sequence is input into the state machine, if the sequence meets the corresponding conditions, the loading state of the target forklift can be switched accordingly. For a detailed explanation of how to process the forklift loading action sequence using a state machine to obtain the target forklift's loading state, please refer to subsequent sections. Figure 6 And its explanations and descriptions.
[0107] It's important to note that state machines offer higher fault tolerance compared to relational databases. For example, consider a sequence of "keep empty, load, keep empty." Under normal circumstances, a forklift cannot remain empty after loading. Therefore, for this erroneous forklift loading sequence, it's often difficult to determine the target forklift's loading status, or the loading status obtained using a relational database is highly likely to be incorrect. However, with a state machine, because the sequence is designed for state transitions, when the forklift loading sequence is incorrect, the condition for state transition is not triggered. In this case, the target forklift's loading status remains the same as the previous loading status. When further action recognition is performed on subsequent image frame sequences to obtain the target forklift's loading action information, the new forklift loading action sequence is used to re-determine the target forklift's loading status, thus obtaining an accurate loading status identification result.
[0108] The technical solution provided in this application obtains a forklift loading action sequence by sorting the forklift loading action information corresponding to each image frame sequence, which facilitates the direct use of the forklift state action sequence to determine the loading status of the target forklift.
[0109] like Figure 6 As shown, Figure 6 This is a flowchart illustrating the fifth embodiment of the loading status identification method provided in this application.
[0110] This embodiment provides a method for switching the state of a target forklift using a forklift loading action sequence and a state machine. Specifically, it includes steps 601-603:
[0111] 601. Input the forklift loading action sequence into a preset state machine to obtain state switching information.
[0112] As described above, the state machine primarily stores the conditions related to state transitions. For ease of understanding, let's take the condition of a forklift switching from empty to fully loaded as an example:
[0113] 1) If the previous loading state of the forklift in the state machine is empty, and the forklift loading action information corresponding to the previous image frame sequence is loading, and the forklift loading action information corresponding to the current image frame sequence is keeping it fully loaded, then switch the current loading state of the forklift in the state machine to fully loaded.
[0114] 2) If the previous loading state of the forklift in the state machine is empty, and the forklift loading action information corresponding to three consecutive image frame sequences is all full load, then the current loading state of the forklift in the state machine switches to full load.
[0115] Thus, by inputting the forklift loading action sequence into a preset state machine, it can be determined whether the forklift loading action sequence meets the conditions for switching from empty to full load or from full load to empty, thereby obtaining state transition information. It should be noted that not meeting the state transition conditions, i.e., maintaining the previous state, can also be considered a type of state transition information. For example, switching from empty to empty or from full load to full load.
[0116] The above only illustrates the forklift status with two states: empty and fully loaded. In fact, based on actual needs, the forklift status can also include other set loading states. In this case, the state machine also needs to include the conditions for switching between other loading states.
[0117] It should be noted that the aforementioned state machine is also pre-stored in the database of the loading status identification device so that the loading status identification device can read it in real time.
[0118] 602. Based on the state switching information, the previous loading state of the target forklift is switched to obtain the switched loading state.
[0119] In this embodiment, after switching the previous loading state of the target forklift using state switching information, the switched loading state can be used as the current loading state of the target forklift. Furthermore, when further identifying the loading state of the target forklift, the current loading state can be considered as the previous loading state of the next loading state.
[0120] It should be noted that the state machine needs to set the initial loading state of the forklift. This initial loading state is determined by the loading action information corresponding to the first image frame sequence in the video stream. Specifically, if the loading action information corresponding to the first image frame sequence is "keeping empty" or "loading," then the initial loading state of the forklift set in the state machine is "empty." If the loading action information corresponding to the first image frame sequence is "keeping fully loaded" or "unloading," then the initial loading state of the forklift set in the state machine is "fully loaded."
[0121] 603, Set the switched loading state to the loading state of the target forklift.
[0122] This embodiment provides a complete implementation method for determining the loading status of a forklift using a state machine. Compared with building an associated database, using a state machine can more effectively improve the fault tolerance rate of the forklift loading status during the identification process, thereby obtaining a more accurate forklift loading status identification result.
[0123] like Figure 7 As shown, Figure 7 This is a flowchart illustrating the sixth embodiment of the loading status identification method provided in this application.
[0124] This embodiment provides a method for processing a video stream to obtain a sequence of multiple image frames. Specifically, it includes steps 701-702:
[0125] 701, The video stream is divided into multiple video stream segments.
[0126] In this embodiment, the loading status recognition device can segment the video stream to obtain multiple video stream segments. There are many specific segmentation rules. For example, the video stream can be segmented based on time, or it can be segmented based on the pause frames of the target forklift in the video stream. That is, when the target forklift is stationary in a certain number of frames, one of those frames is used as the segmentation point to segment the video stream.
[0127] 702. For each video stream segment, the video stream segment is sampled to obtain the image frame sequence of the video stream segment.
[0128] In this embodiment, for each video stream segment, a sequence of image frames can be obtained by sampling and extracting several image frames from it. It should be noted that to avoid losing some motion feature information of the target forklift in the video stream, the sampling frequency of the video stream should not be too low; conversely, to reduce computational load, the sampling frequency should also not be too high. As an optional approach, a sampling frequency of 0.5 seconds can typically be used to sample video stream segments.
[0129] As an optional approach, after sampling the video stream segments and extracting several image frames, to further reduce the computational load, each extracted image frame will be resampled. For details, please refer to [link to relevant documentation]. Figure 8 And its explanations and descriptions.
[0130] In this embodiment, by dividing the video stream into multiple video stream segments according to specific rules, and sampling and extracting several image frames from each video stream segment to form an image frame sequence, multiple image frame sequences corresponding to the video stream can be obtained, which can be further used for subsequent identification of loading action information and loading status.
[0131] like Figure 8 As shown, Figure 8 This is a flowchart illustrating the seventh embodiment of the loading status identification method provided in this application.
[0132] Considering the high real-time requirements for status recognition during the real-time loading status identification of the target forklift, this application proposes a technical solution to improve the real-time performance of status recognition by cropping image frames during the generation of the image frame sequence to reduce subsequent computational load. Specifically, this includes steps 801-803:
[0133] 801, sample the video stream segment to obtain multiple sampled images.
[0134] In this embodiment, the image frame sequence described in step 702 can be obtained by sequentially arranging the multiple sampled images obtained by sampling video stream segments. For specific sampling instructions, please refer to the explanation of step 701 above.
[0135] 802. For each sampled image, the sampled image is cropped according to the target detection result of the sampled image to obtain a cropped sampled image.
[0136] In this embodiment, by performing target detection on the sampled image, the portion of the sampled image related to the target forklift can be detected. Then, the sampled image is cropped using the target detection results, retaining only the portion of the sampled image related to the target forklift while removing complex background areas. This reduces the computational load for subsequent behavior recognition of image frame sequences and improves real-time performance.
[0137] In this embodiment, there are many ways to perform object detection on the sampled images, which will not be elaborated here. For example, the forklift in each frame of the sampled image can be detected using the YOLOv3 object detector network, which has the advantages of high speed and small model size, and the coordinates of the forklift in each frame can be obtained. Then, the coordinates of the forklift in each frame can be used to crop the sampled image to obtain the cropped sampled image.
[0138] 803. Generate an image frame sequence of the video stream segment based on each of the cropped sampled images.
[0139] In this embodiment, further considering that the target forklift occupies different proportions in each frame of the sampled image, meaning that the cropped sampled images are often of different sizes, the cropped sampled images can be scaled proportionally and filled with gray pixel blocks to form a 224x224 image, which is then arranged to obtain an image frame sequence, thereby ensuring that the size of each image in the generated image frame sequence is the same.
[0140] This embodiment performs target detection and cropping on the sampled image frames to retain the parts of the sampled image related to the target forklift, thereby reducing the amount of subsequent calculations and effectively improving the real-time performance of state recognition.
[0141] like Figure 9 As shown, Figure 9 This is a schematic diagram of an embodiment of the loading status identification device provided in this application.
[0142] To better implement the loading status identification method in the embodiments of this application, based on the loading status identification method, the embodiments of this application also provide a loading status identification device, which includes:
[0143] The video stream acquisition module 901 is used to acquire the video stream corresponding to the target forklift.
[0144] The image frame sequence generation module 902 is used to generate multiple image frame sequences based on the video stream.
[0145] The action recognition module 903 is used to determine the forklift loading action information corresponding to each image frame sequence based on the multiple images contained in the image frame sequence.
[0146] The status recognition module 904 is used to determine the loading status of the target forklift based on the forklift loading action information corresponding to each of the image frame sequences.
[0147] In some embodiments of this application, the above-mentioned action recognition module includes a feature extraction submodule, a confidence calculation submodule, and a loading action determination submodule, wherein:
[0148] The feature extraction submodule is used to input multiple frames of images contained in the image frame sequence into a preset behavior recognition model to obtain the load change feature information of the target forklift;
[0149] The confidence calculation submodule is used to determine the classification confidence of each preset target forklift loading action information in the behavior recognition model based on the load change feature information.
[0150] The loading action determination submodule is used to set the target forklift loading action information corresponding to the maximum classification confidence in the classification confidence as the forklift loading action information corresponding to the image frame sequence.
[0151] In some embodiments of this application, the above-mentioned loading action determination submodule includes a first loading action determination unit and a second loading action determination unit, wherein:
[0152] The first loading action determination unit is used to set the target forklift loading action information corresponding to the maximum classification confidence in the classification confidence to the forklift loading action information corresponding to the image frame sequence if the maximum classification confidence in the classification confidence is greater than a preset confidence threshold.
[0153] The second loading action determination unit is used to set the forklift loading action information corresponding to the previous image frame sequence of the image frame sequence as the forklift loading action information corresponding to the image frame sequence if the maximum classification confidence in the classification confidence is less than or equal to a preset confidence threshold.
[0154] In some embodiments of this application, the state recognition module includes a sorting submodule and a state recognition submodule, wherein:
[0155] The sorting submodule is used to sort the forklift loading action information corresponding to each of the image frame sequences according to the order of each image frame sequence, so as to obtain the forklift loading action sequence.
[0156] The status recognition submodule is used to determine the loading status of the target forklift based on the forklift loading action sequence.
[0157] In some embodiments of this application, the state recognition submodule includes a state machine processing unit, a state switching unit, and a state setting unit, wherein:
[0158] The state machine processing unit is used to input the forklift loading action sequence into a preset state machine to obtain state switching information;
[0159] A state switching unit is used to switch the previous loading state of the target forklift according to the state switching information to obtain the switched loading state.
[0160] A status setting unit is used to set the switched loading status to the loading status of the target forklift.
[0161] In some embodiments of this application, the above-mentioned image frame sequence generation module includes a partitioning submodule and a sampling submodule, wherein:
[0162] The segmentation submodule is used to segment the video stream to obtain multiple video stream segments;
[0163] The sampling submodule is used to sample each video stream segment to obtain an image frame sequence of the video stream segment.
[0164] In some embodiments of this application, the above-mentioned sampling submodule includes a sampling unit, a cropping unit, and an image frame sequence generation unit, wherein:
[0165] A sampling unit is used to sample the video stream segments to obtain multiple sampled images;
[0166] The cropping unit is used to crop the sampled image according to the target detection result of the sampled image for each sampled image to obtain the cropped sampled image;
[0167] The image frame sequence generation unit is used to generate an image frame sequence of the video stream segment based on each of the cropped sampled images.
[0168] This invention also provides a loading status identification device, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of an embodiment of the loading status identification device provided in this application.
[0169] The loading status identification device includes a memory, a processor, and a loading status identification program stored in the memory and executable on the processor. When the processor executes the loading status identification program, it implements the steps in the loading status identification method of any embodiment.
[0170] Specifically, the loading status identification device may include components such as a processor 1001 with one or more processing cores, a memory 1002 with one or more storage media, a power supply 1003, and an input unit 1004. Those skilled in the art will understand that... Figure 10 The structure of the loading status identification device shown does not constitute a limitation on the loading status identification device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:
[0171] The processor 1001 is the control center of the loading status identification device. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1002, and by calling data stored in the memory 1002, thereby providing overall monitoring of the loading status identification device. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1001.
[0172] The memory 1002 can be used to store software programs and modules. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the usage of the device according to the loading status. In addition, the memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.
[0173] The loading status identification device also includes a power supply 1003 that supplies power to the various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1003 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0174] The loading status recognition device may also include an input unit 1004, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0175] Although not shown, the loading status identification device may also include a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 1001 in the loading status identification device loads the executable files corresponding to the processes of one or more application programs into the memory 1002 according to the following instructions, and the processor 1001 runs the application programs stored in the memory 1002, thereby implementing the steps in any of the loading status identification methods provided in the embodiments of this application. For example Figure 2 As shown:
[0176] Obtain the video stream corresponding to the target forklift;
[0177] Generate multiple image frame sequences based on the video stream;
[0178] For each image frame sequence, the forklift loading action information corresponding to the image frame sequence is determined based on the multiple images contained in the image frame sequence.
[0179] The loading status of the target forklift is determined based on the forklift loading action information corresponding to each of the image frame sequences.
[0180] Therefore, embodiments of the present invention provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. A computer program is stored on the computer-readable storage medium, and a load state identification program is stored on the computer-readable storage medium. When the load state identification program is executed, it implements the steps of any of the load state identification methods provided in the embodiments of this application. For example, Figure 2 As shown:
[0181] Obtain the video stream corresponding to the target forklift;
[0182] Generate multiple image frame sequences based on the video stream;
[0183] For each image frame sequence, the forklift loading action information corresponding to the image frame sequence is determined based on the multiple images contained in the image frame sequence.
[0184] The loading status of the target forklift is determined based on the forklift loading action information corresponding to each of the image frame sequences.
[0185] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.
[0186] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.
[0187] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0188] The above provides a detailed description of a loading status identification method provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for identifying loading status, characterized in that, include: Obtain the video stream corresponding to the target forklift; Generate multiple image frame sequences based on the video stream; For each image frame sequence, forklift loading action information corresponding to the image frame sequence is determined based on the multiple images contained in the image frame sequence; the forklift loading action information includes at least one of keeping empty, keeping loaded, loading, and unloading; The loading status of the target forklift is determined based on the forklift loading action information corresponding to each of the image frame sequences. Based on the multiple images contained in the image frame sequence, determine the forklift loading action information corresponding to the image frame sequence, including: The multiple frames of images contained in the image frame sequence are input into a preset behavior recognition model to obtain the load change feature information of the target forklift; Based on the load change feature information, determine the classification confidence level corresponding to each preset target forklift loading action information in the behavior recognition model; Set the target forklift loading action information corresponding to the highest classification confidence in the classification confidence as the forklift loading action information corresponding to the image frame sequence; Determining the loading status of the target forklift based on the forklift loading action information corresponding to each of the image frame sequences includes: The forklift loading action information corresponding to each image frame sequence is sorted according to the order of each image frame sequence to obtain the forklift loading action sequence. The forklift loading action sequence is input into a preset state machine to obtain state switching information; The previous loading state of the target forklift is switched according to the state switching information to obtain the switched loading state; Set the switched loading state to the loading state of the target forklift.
2. The method according to claim 1, characterized in that, Setting the target forklift loading action information corresponding to the highest classification confidence score in the classification confidence scores as the forklift loading action information corresponding to the image frame sequence includes: If the maximum classification confidence score in the classification confidence scores is greater than a preset confidence threshold, then the target forklift loading action information corresponding to the maximum classification confidence score in the classification confidence scores is set as the forklift loading action information corresponding to the image frame sequence. If the maximum classification confidence score is less than or equal to a preset confidence threshold, then the forklift loading action information corresponding to the previous image frame sequence of the image frame sequence is set as the forklift loading action information corresponding to the image frame sequence.
3. The method according to any one of claims 1 to 2, characterized in that, The step of generating multiple image frame sequences based on the video stream includes: The video stream is divided into multiple video stream segments; For each video stream segment, the video stream segment is sampled to obtain the image frame sequence of the video stream segment.
4. The method according to claim 3, characterized in that, The step of sampling the video stream segment to obtain the image frame sequence of the video stream segment includes: The video stream segment is sampled to obtain multiple sampled images; For each sampled image, the sampled image is cropped according to the target detection result of the sampled image to obtain the cropped sampled image; Based on the cropped sampled images, an image frame sequence of the video stream segment is generated.
5. A loading status identification device, characterized in that, include: The video stream acquisition module is used to acquire the video stream corresponding to the target forklift; An image frame sequence generation module is used to generate multiple image frame sequences based on the video stream; The action recognition module is used to determine the forklift loading action information corresponding to each image frame sequence based on the multiple images contained in the image frame sequence; the forklift loading action information includes at least one of keeping empty, keeping loaded, loading, and unloading. The status recognition module is used to determine the loading status of the target forklift based on the forklift loading action information corresponding to each of the image frame sequences; Based on the multiple images contained in the image frame sequence, determine the forklift loading action information corresponding to the image frame sequence, including: The multiple frames of images contained in the image frame sequence are input into a preset behavior recognition model to obtain the load change feature information of the target forklift; Based on the load change feature information, determine the classification confidence level corresponding to each preset target forklift loading action information in the behavior recognition model; Set the target forklift loading action information corresponding to the highest classification confidence in the classification confidence as the forklift loading action information corresponding to the image frame sequence; Determining the loading status of the target forklift based on the forklift loading action information corresponding to each of the image frame sequences includes: The forklift loading action information corresponding to each image frame sequence is sorted according to the order of each image frame sequence to obtain the forklift loading action sequence. The forklift loading action sequence is input into a preset state machine to obtain state switching information; The previous loading state of the target forklift is switched according to the state switching information to obtain the switched loading state; Set the switched loading state to the loading state of the target forklift.
6. A loading status identification device, characterized in that, The loading status identification device includes a processor, a memory, and a loading status identification program stored in the memory and executable on the processor. The processor executes the loading status identification program to implement the steps of the loading status identification method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a loading status identification program, which is executed by a processor to implement the steps of the loading status identification method according to any one of claims 1 to 4.