Vehicle hopper manned identification method and device and storage medium

Abstract

The invention discloses a car hopper manned identification method and device and a storage medium, relates to the technical field of image processing and identification, and discloses a car hopper manned identification method comprising the following steps: detecting N video frames where a target car is located from a plurality of video frames of a collected video stream; scoring the N video frames according to at least one of the image quality and the dynamic information of the N video frames to obtain a scoring result; screening at least two target video frames from the N video frames according to the scoring result; respectively inputting the at least two target video frames into a pre-trained identification model, and identifying an identification result respectively corresponding to each target video frame by the identification model; and based on the identification results corresponding to the at least two target video frames, judging whether the hopper of the target vehicle carries people or not. According to the invention, the multiple video frames are utilized to comprehensively identify whether the car hopper carries people, so that the identification accuracy in a complex and changeable traffic scene is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing and recognition technology, and in particular to a method, device and storage medium for identifying people in a vehicle bed. Background Art

[0002] As the number of motor vehicles on urban roads continues to rise, illegal driving behaviors of motor vehicles, especially illegal carrying of passengers in vehicle cargo boxes, are increasingly becoming a major hidden danger affecting traffic safety. In the face of this challenge, the development of intelligent transportation systems is particularly important. Among them, vehicle cargo occupant recognition technology, as one of the key technologies, has achieved accurate capture and reporting of illegal driving behaviors, providing a strong basis for traffic management departments.

[0003] At present, traditional truck bed passenger recognition technology mostly relies on single random frame processing, but this method usually seems to be powerless when faced with complex and changeable traffic scenes. Since the target vehicle in the video will experience multiple consecutive frames from entering the field of view to leaving, the target state may be diverse due to occlusion, angle changes (such as back, sideways) and other factors. Single frame processing is difficult to fully capture these changes, thus affecting the accuracy and stability of recognition.

[0004] Therefore, how to provide a method for identifying passengers in a truck bed that can adapt to complex and changeable traffic scenarios is an urgent problem to be solved.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a method, device and storage medium for identifying passengers in a vehicle bed, aiming to provide a method for identifying passengers in a vehicle bed that can adapt to complex and changeable traffic scenarios.

[0007] To achieve the above purpose, the present application proposes a method for identifying passengers in a vehicle compartment, the method comprising:

[0008] Detecting N video frames where the target vehicle is located from multiple video frames of the collected video stream; wherein N is an integer greater than 2;

[0009] Scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result; wherein the scoring result represents the reliability of using each video frame to identify the passenger in the vehicle compartment of the target vehicle;

[0010] Selecting at least two target video frames from the N video frames according to the scoring results;

[0011] Inputting the at least two target video frames into a pre-trained recognition model respectively, and using the recognition model to identify recognition results corresponding to the target video frames respectively;

[0012] Based on the recognition results respectively corresponding to the at least two target video frames, it is determined whether the vehicle compartment of the target vehicle is carrying people.

[0013] In one embodiment, scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result includes:

[0014] Performing an image quality score on each of the N video frames to obtain a target quality comprehensive score corresponding to each video frame;

[0015] Performing a dynamic information score on each of the N video frames to obtain a dynamic information score corresponding to each video frame;

[0016] Determine a scoring result corresponding to each video frame based on at least one of a target quality comprehensive score and a dynamic information score corresponding to each video frame;

[0017] The step of selecting at least two target video frames from the N video frames according to the scoring results includes:

[0018] At least two target video frames are selected from the N video frames according to the order of the scoring results corresponding to the video frames from high to low.

[0019] In one embodiment, the N video frames are a plurality of video frames that are continuous in chronological order, and the step of performing a dynamic information score on each of the N video frames to obtain a dynamic information score corresponding to each video frame includes:

[0020] The following steps are performed for each of the N video frames:

[0021] Extracting key dynamic information corresponding to the video frame; wherein the key dynamic information includes at least one of optical flow information and feature point motion information, the optical flow information represents the motion of the feature point in the video frame between two consecutive frames, and the feature point motion information represents the position change of the feature point in the video frame in the time series;

[0022] Based on the key dynamic information, calculate relevant parameters of the feature point in the video frame; wherein the relevant parameters include at least one of the motion intensity change rate, the complex intensity of the moving path, the speed change amount and the acceleration change amount, the motion intensity change rate represents the change rate between the total displacement of the feature point in the video frame or the modulus of the optical flow vector and the previous video frame, the complex intensity of the moving path represents the ratio of the moving path distance of the feature point in the video frame to the straight-line distance between the starting point and the end point of the moving path, the speed change amount represents the speed change amount of the feature point in the video frame between the m video frames before and after the video frame, and the acceleration change amount represents the acceleration change amount of the feature point in the video frame between the m video frames before and after the video frame, where m is an integer greater than 2;

[0023] Determining a candidate score of dynamic information corresponding to the video frame based on relevant parameters of the feature points in the video frame;

[0024] The dynamic information score of the video frame is determined based on the dynamic information candidate scores respectively corresponding to the video frame and the previous video frame.

[0025] In one embodiment, the step of performing image quality scoring on each of the N video frames to obtain a target quality comprehensive score corresponding to each video frame includes:

[0026] The following steps are performed for each of the N video frames:

[0027] Evaluating the image clarity of the video frame to obtain a first score;

[0028] Evaluating the completeness of the target vehicle in the video frame to obtain a second score;

[0029] evaluating the prominence of the target vehicle in the video frame to obtain a third score;

[0030] Based on at least two of the first score and the corresponding first weight, the second score and the corresponding second weight, and the third score and the corresponding third weight, a target quality comprehensive score corresponding to the video frame is calculated.

[0031] In one embodiment, the step of evaluating the image clarity of the video frame to obtain a first score includes:

[0032] Using the Sobel edge detection method, the gradient of the grayscale value of the pixel in the video frame is calculated; wherein the gradient of the grayscale value of the pixel in the video frame is used to characterize the image clarity;

[0033] Obtaining a first candidate score for the video frame based on a mean value of a gradient of pixel grayscale values ​​in the video frame;

[0034] Using the Laplacian gradient detection method, the second-order derivative of the grayscale value of the pixel in the video frame is calculated; wherein the second-order derivative of the grayscale value of the pixel in the video frame is used to characterize the image clarity;

[0035] Obtaining a second candidate score for the video frame by calculating a Laplace gradient variance based on a second-order derivative of a pixel grayscale value in the video frame;

[0036] A first score is determined based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight.

[0037] In one embodiment, the step of determining the first score based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight, comprises:

[0038] Obtaining a third candidate score for the video frame based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight;

[0039] Performing K-Medoids clustering on the third candidate scores corresponding to the N video frames respectively to obtain K clusters; wherein the difference between the third candidate scores corresponding to the video frames in the same cluster is less than the first value; wherein K represents the number of categories, and K is an integer greater than 1;

[0040] Sorting the K clusters according to the central value of the third candidate score in each cluster from small to large to obtain ordered K clusters;

[0041] For each of the K ordered clusters, based on the sequence number of the cluster in the K ordered clusters and K, a clarity score of the cluster is determined as a first score corresponding to all video frames in the cluster.

[0042] In one embodiment, the step of evaluating the completeness of the target vehicle in the video frame to obtain a second score includes:

[0043] Inputting the video frame into a pre-trained target detection model, and using the target detection model to evaluate the integrity of the target vehicle in the video frame to obtain a second score;

[0044] The training set used when training the target detection model includes samples with occluded targets and samples with incomplete targets.

[0045] In one embodiment, the step of evaluating the prominence of the target vehicle in the video frame to obtain a third score includes:

[0046] The video frame is input into a preset saliency recognition model, and the saliency recognition model evaluates the saliency of the target area in the video frame, and outputs the grayscale value corresponding to each pixel in the target area; wherein the target area is the area of ​​the target vehicle in the video frame, and the grayscale value corresponding to each pixel in the target area represents the saliency strength of each pixel in the video frame;

[0047] A third score is determined based on the sum of the grayscale values ​​corresponding to all pixels in the target area.

[0048] In one embodiment, before the step of scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result, the method further includes:

[0049] For each of the N video frames, the following steps are performed:

[0050] Detect and select the framed area where the target vehicle is located;

[0051] The area after the framed area is expanded according to a preset scaling ratio is used as the target detection area of ​​the video frame; wherein the scaling ratio is greater than 1;

[0052] The step of scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result comprises:

[0053] According to at least one of the image quality and dynamic information of the N video frames, the target detection area of ​​each of the N video frames is scored respectively to obtain a scoring result.

[0054] In one embodiment, the recognition model includes a vehicle type recognition module, a truck loading recognition module and a truck loading passenger recognition module, and the recognition result includes a first result output by the vehicle type recognition module, a second result output by the truck loading recognition module and a third result output by the truck loading passenger recognition module;

[0055] The step of inputting the at least two target video frames into a pre-trained recognition model respectively, and using the recognition model to identify the recognition results corresponding to the target video frames respectively, comprises:

[0056] For each of the at least two target video frames, the following steps are performed:

[0057] Inputting the target video frame into the vehicle type recognition module, and having the vehicle type recognition module recognize the vehicle type of the target vehicle from the target video frame as the first result;

[0058] The target video frame is input into the bucket loading recognition module, and the bucket loading recognition module recognizes the bucket loading state of the target vehicle from the target video frame as the second result; wherein the bucket loading state includes the bucket being loaded or the bucket being unloaded;

[0059] The target video frame is input into the vehicle load recognition module, and the vehicle load recognition module recognizes the vehicle load state of the target vehicle from the target video frame as the third result; wherein the vehicle load state includes the vehicle load being loaded or the vehicle load being unloaded;

[0060] Among them, the vehicle type recognition module, the truck bed loading recognition module and the truck bed passenger recognition module are trained based on a multi-task learning method.

[0061] In one embodiment, before the step of inputting the at least two target video frames into a pre-trained recognition model respectively, and allowing the recognition model to identify the recognition results corresponding to each of the target video frames respectively, the method further includes:

[0062] Using a pre-trained shared feature extractor to extract shared features in a training set, and building a shared backbone network based on the shared features;

[0063] Using the training set to train a preset first branch network, a second branch network and a third branch network respectively;

[0064] Based on the shared backbone network and the trained first branch network, a vehicle type recognition candidate network is constructed; based on the shared backbone network and the trained second branch network, a vehicle loading recognition candidate network is constructed; based on the shared backbone network and the trained third branch network, a vehicle loading candidate network is constructed;

[0065] Using a preset multi-task loss function, the vehicle type recognition candidate network, the truck bed loading recognition candidate network and the truck bed passenger candidate network are trained respectively to obtain the vehicle type recognition module, the truck bed loading recognition module and the truck bed passenger recognition module; wherein the multi-task loss function is the weighted sum of the loss functions of the vehicle type recognition candidate network, the truck bed loading recognition candidate network and the truck bed passenger candidate network.

[0066] In one embodiment, the step of judging whether the bucket of the target vehicle is carrying people based on the recognition results corresponding to the at least two target video frames respectively includes:

[0067] For each of the at least two target video frames, determining whether a first result corresponding to the target video frame is a target vehicle type with a truck bucket;

[0068] In the case where it is determined that the first result corresponding to the target video frame is the target vehicle type, it is determined whether the second result corresponding to the target video frame is that the bucket is loaded;

[0069] In the case where it is determined that the second result corresponding to the target video frame is that the bucket is loaded, the third result corresponding to the target video frame is taken as a valid result;

[0070] Based on the valid results corresponding to the at least two target video frames, it is determined whether the vehicle compartment of the target vehicle is carrying people.

[0071] In one embodiment, the step of determining whether the bucket of the target vehicle is carrying people based on the valid results corresponding to the at least two target video frames includes:

[0072] When it is determined that the number of corresponding valid results in the at least two target video frames is an odd number, comparing a first number of valid results indicating that the vehicle bucket is loaded with people and a second number of valid results indicating that the vehicle bucket is not loaded with people, if the first number is greater than the second number, determining that the vehicle bucket of the target vehicle is loaded with people; if the first number is less than the second number, determining that the vehicle bucket of the target vehicle is not loaded with people;

[0073] When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence among the valid results is greater than or equal to the second value, if the valid result with the highest confidence indicates that the bucket is loaded with people, then it is determined that the bucket of the target vehicle is loaded with people; if the valid result with the highest confidence indicates that the bucket is not loaded with people, then it is determined that the bucket of the target vehicle is not loaded with people;

[0074] When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence level among the valid results is less than a second value, it is determined that the bucket of the target vehicle is unmanned.

[0075] In addition, to achieve the above purpose, the present application also proposes a vehicle loading person identification device, the device comprising:

[0076] A detection module, used to detect N video frames where the target vehicle is located from multiple video frames of the collected video stream; wherein N is an integer greater than 2;

[0077] A scoring module, used for scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result; wherein the scoring result represents the reliability of using each video frame to identify the passenger in the vehicle compartment of the target vehicle;

[0078] A screening module, used for screening at least two target video frames from the N video frames according to the scoring result;

[0079] A recognition module, used for inputting the at least two target video frames into a pre-trained recognition model respectively, and the recognition model recognizes the recognition results corresponding to the target video frames respectively;

[0080] The judgment module is used to judge whether the vehicle compartment of the target vehicle is carrying people based on the recognition results corresponding to the at least two target video frames.

[0081] In addition, to achieve the above-mentioned purpose, the present application also proposes a device for identifying passengers in a vehicle bed, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for identifying passengers in a vehicle bed as described above.

[0082] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the method for identifying passengers in the vehicle compartment as described above are implemented.

[0083] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method for identifying passengers in a vehicle compartment as described above.

[0084] One or more technical solutions proposed in this application have at least the following technical effects:

[0085] 1) Compared with the related art that only relies on a single random frame for identification, this application uses multiple video frames to comprehensively identify whether the truck bed is loaded with people, which improves the accuracy of identification in complex and changeable traffic scenes, that is, it can adapt to complex and changeable traffic scenes;

[0086] 2) Compared with the frame-by-frame processing strategy adopted in the related art to adapt to complex and changing scenes, which brings about problems such as excessive consumption of computing resources and low detection efficiency, the present application scores multiple video frames respectively according to at least one of the image quality and dynamic information of N video frames, and selects at least two target video frames for subsequent recognition according to the scoring results, which can effectively reduce computing resources and improve detection efficiency while ensuring recognition accuracy;

[0087] 3) The present application also inputs the screened at least two target video frames into the pre-trained recognition model respectively, so that the recognition model can more efficiently and accurately identify the recognition results corresponding to each target video frame, and finally judge whether the cargo box of the target vehicle is loaded with people based on these recognition results. That is, the recognition results of multiple video frames are combined to finally judge whether the cargo box of the target vehicle is loaded with people, which effectively improves the accuracy of identifying whether there are people in the cargo box. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0089] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0090] Figure 1 One of the flow charts of the method for identifying passengers in a vehicle compartment provided by this application;

[0091] Figure 2 The second flowchart of the method for identifying passengers in a vehicle compartment provided by the present application;

[0092] Figure 3 The third flowchart of the method for identifying passengers in a vehicle compartment provided by this application;

[0093] Figure 4 The fourth flowchart of the method for identifying a person in a vehicle compartment provided by the present application;

[0094] Figure 5 A schematic diagram of N frames from the appearance to the disappearance of a target in the method for identifying a person in a vehicle compartment provided by the present application;

[0095] Figure 6 The fifth flowchart of the method for identifying passengers in a vehicle compartment provided by the present application;

[0096] Figure 7 The sixth flow chart of the method for identifying a person in a vehicle compartment provided by the present application;

[0097] Figure 8 A schematic diagram of multi-task learning in a deep neural network in the method for identifying passengers in a vehicle compartment provided in this application;

[0098] Fig. 9 The seventh flowchart of the method for identifying a person in a vehicle compartment provided by the present application;

[0099] Fig.10 The eighth flowchart of the method for identifying a person in a vehicle compartment provided by the present application;

[0100] Fig.11 A schematic diagram of the flow of the optimal frame capture module in the method for identifying passengers in a vehicle compartment provided in this application;

[0101] Fig.12 A schematic diagram of the flow of the result fusion module in the method for identifying passengers in a vehicle compartment provided in this application;

[0102] Fig.13 A schematic diagram of the structure of the vehicle loading person identification device provided in this application;

[0103] Fig.14 A schematic diagram of the structure provided for the vehicle bed passenger identification device of the present application.

[0104] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0105] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0106] Unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by technicians in the technical field to which this application belongs; the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0107] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0108] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0109] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0110] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0111] In the related technologies, the following two solutions are mainly used for the identification of passengers in the truck bed:

[0112] 1) Traditional truck bed passenger recognition technology mostly relies on single random frame processing, which is incapable of coping with complex and ever-changing traffic scenes. Since the target vehicle in the video will experience multiple consecutive frames from entering the field of view to leaving, the target state may be diverse due to factors such as occlusion and angle changes (such as back and sideways). Single random frame processing is difficult to fully capture these changes, thus affecting the accuracy and stability of recognition.

[0113] 2) A frame-by-frame processing strategy can also be used for passenger identification in the truck bed. Although the frame-by-frame processing strategy attempts to improve the recognition accuracy by increasing the number of analysis frames, it brings about problems such as excessive consumption of computing resources, low detection efficiency, and difficulty in effectively integrating results. This strategy performs a large number of repeated detections and analyses on the same target, which not only increases the system burden, but also may lead to unstable recognition results.

[0114] In response to the above problems, the present application provides a method for identifying passengers in a truck bed, which aims to screen and extract the "three golden frames" with the highest quality and the richest information in the target vehicle's motion path by introducing a dynamic multi-frame optimization algorithm based on a multi-dimensional quality scoring system, and use them as target video frames for subsequent identification. These three frames of images can effectively cover the various state changes of the target vehicle, avoiding the problem of missing information caused by single-frame processing. While ensuring recognition accuracy, unnecessary computational burden is minimized. In addition, the present application also introduces an intelligent collaborative multi-task learning framework and an effective result fusion mechanism. The framework improves its adaptability to complex and changing scenarios through the learning process of the three tasks of combined vehicle type recognition, truck bed loading status recognition, and truck bed passenger recognition. At the same time, the result fusion mechanism is responsible for integrating the recognition results output by each task to ensure the accuracy and reliability of the final recognition result.

[0115] It should be noted that the optimal frame capture in the relevant technology refers to selecting the frame that best meets specific conditions (such as target clarity, integrity, etc.) from continuous video frames for capture or processing according to a certain scoring mechanism or algorithm in the video processing or monitoring system. Its application in this application is reflected in: by introducing a dynamic frame selection mechanism based on video frame quality and inter-frame differences, only the three frames (or multiple frames) with the highest quality from the appearance to the disappearance of the target are selected for subsequent recognition analysis to improve the real-time performance and accuracy of the system.

[0116] It should also be noted that the above multi-task learning is a machine learning paradigm that allows a model to learn to perform multiple related but not identical tasks at the same time. The core of this method is that by sharing representations or weights, different tasks can promote each other in the learning process, thereby improving the overall learning efficiency and the generalization ability of the model. This application improves the overall recognition performance by simultaneously learning the three tasks of vehicle type recognition, truck loading status recognition, and truck occupant recognition.

[0117] It should also be noted that the above-mentioned result fusion is a commonly used technology in machine learning, data analysis and data science. Its main purpose is to merge the prediction or analysis results from different models, different methods or different data sources to produce a more accurate and comprehensive result; result fusion can be achieved in a variety of ways, including but not limited to the averaging method, voting method, etc. This application combines the analysis results of the cargo compartment load of three pictures by voting method combined with confidence ranking method, so as to ensure the accuracy and rationality of the final output result.

[0118] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a vehicle load passenger identification device, etc. The following takes the vehicle load passenger identification device as an example to illustrate this embodiment and the following embodiments.

[0119] This embodiment and the following embodiments are described in detail below.

[0120] The present application embodiment provides a method for identifying a person in a vehicle compartment, referring to Figure 1 , Figure 1 This is one of the flow charts of the method for identifying passengers in a vehicle bed provided in this application.

[0121] In this embodiment, the method for identifying a person in a vehicle bucket includes steps S10 to S50:

[0122] Step S10, detecting N video frames where the target vehicle is located from the multiple video frames of the collected video stream;

[0123] Wherein, N is an integer greater than 2.

[0124] It should be noted that in the video structured capture application, a target will appear in multiple frames from its appearance to its disappearance, and there are various target states, such as being blocked, facing away, facing sideways, facing forward, or being smaller or larger. In order to capture clear targets and ensure the accuracy of subsequent analysis results, this application proposes to obtain multiple video frames based on the video stream, and then capture multiple video frames with higher reliability as the input of the subsequent recognition model, which can make up for the inaccurate recognition of a single frame due to occlusion, blur, etc.

[0125] It should also be noted that the sampling system can collect video streams in real time through cameras and other acquisition devices. The detection module detects the target vehicles in the video frames, and then the tracking module uniquely identifies each target vehicle, which facilitates the subsequent identification of results related to the target vehicle. There is no need to repeatedly detect the target vehicles in each video frame, which effectively reduces the amount of detection.

[0126] Step S20, scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result;

[0127] The scoring result represents the reliability of identifying the occupants in the target vehicle using each video frame.

[0128] Step S30: screening at least two target video frames from the N video frames according to the scoring results.

[0129] It should be noted that the number of screened target video frames can be set according to actual conditions, for example, it can be set to 3, but is usually less than N, and this application does not limit it here.

[0130] Step S40: input the at least two target video frames into pre-trained recognition models respectively, and use the recognition models to identify recognition results corresponding to the target video frames respectively.

[0131] Step S50, based on the recognition results corresponding to the at least two target video frames, determine whether the vehicle compartment of the target vehicle is carrying people.

[0132] Specifically, the present application first detects N video frames where the target vehicle is located from multiple video frames of the collected video stream, and then uses a scoring mechanism to screen some target video frames for subsequent identification. Specifically, the N video frames are first scored according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result, and the reliability of subsequent identification of people in the cargo bed using each video frame is characterized by the scoring result. Then, at least two target video frames are obtained from them according to the scoring result for subsequent identification. Specifically, the at least two target video frames are respectively input into a pre-trained recognition model for identification, and the recognition model identifies and outputs the recognition results corresponding to each target video frame, and then based on the recognition results corresponding to the at least two target video frames, a comprehensive judgment is made as to whether the cargo bed of the target vehicle carries people.

[0133] The present embodiment provides a method for identifying whether a vehicle bed is carrying people. Compared with the related art that relies only on a single random frame for identification, the present application uses multiple video frames to comprehensively identify whether the vehicle bed is carrying people, thereby improving the accuracy of identification in complex and changeable traffic scenes, that is, it can adapt to complex and changeable traffic scenes. In addition, compared with the frame-by-frame processing strategy adopted in the related art to adapt to complex and changeable scenes, which brings problems such as excessive consumption of computing resources and low detection efficiency, the present application scores multiple video frames respectively according to at least one of the image quality and dynamic information of N video frames, and selects at least two target video frames for subsequent identification according to the scoring results, while ensuring the accuracy of identification, it can effectively reduce computing resources and improve detection efficiency. In addition, the present application also inputs the at least two selected target video frames into the pre-trained recognition model respectively, so that the recognition model can more efficiently and accurately identify the recognition results corresponding to each target video frame, and finally judges whether the vehicle bed of the target vehicle is carrying people based on these recognition results, that is, it finally judges whether the vehicle bed of the target vehicle is carrying people based on the recognition results of multiple video frames, which effectively improves the accuracy of identifying whether the vehicle bed is carrying people.

[0134] Based on this application Figure 1 Corresponding embodiments, in this embodiment, the same or similar contents as those in the above embodiments can be referred to the above description, and will not be described in detail later. On this basis, this application provides a specific implementation method for scoring each video frame, please refer to Figure 2 , Figure 2 The second flow chart of the method for identifying a person in a vehicle compartment provided by the present application, step S20 includes steps S201 to S203:

[0135] Step S201 , performing image quality scoring on each of the N video frames to obtain a target quality comprehensive score corresponding to each video frame.

[0136] Step S202 , performing dynamic information scoring on each of the N video frames to obtain a dynamic information score corresponding to each video frame.

[0137] Step S203: Determine a scoring result corresponding to each video frame based on at least one of the target quality comprehensive score and the dynamic information score corresponding to each video frame.

[0138] It should be noted that the scoring result corresponding to each video frame can be determined based on the above-mentioned target quality comprehensive score and / or the above-mentioned dynamic information score according to the actual situation. For example, the dynamic information score usually needs to be analyzed for several consecutive video frames. If there are fewer consecutive video frames in the above-mentioned N video frames, the scoring result can be determined by only using the target quality comprehensive score; if there are more consecutive video frames in the above-mentioned N video frames, the dynamic information score can be considered to be introduced to jointly score each video frame, which can further improve the objectivity and accuracy of the score.

[0139] Step S30 includes step S30-1:

[0140] Step S30 - 1 , selecting at least two target video frames from the N video frames in descending order of the scoring results corresponding to the video frames.

[0141] It should be noted that since the scoring results can represent the reliability of using each video frame to identify the occupants in the target vehicle's cargo box, multiple target frames can be selected for subsequent identification in descending order of the scoring results, that is, video frames with higher recognition reliability are selected as target video frames for subsequent identification.

[0142] It should also be noted that the present application does not limit the order of execution between the above-mentioned steps S201 and S202, that is, the image quality scoring can be performed first, or the dynamic information scoring can be performed first, or the image quality scoring and the dynamic information scoring can be performed at the same time. The order shown in the figure is only for illustration, and the present application does not impose any limitation on this.

[0143] In an embodiment of the present application, when scoring video frames, reference can be made to both the image quality score and the dynamic information score to comprehensively determine target video frames with higher scores, which helps to score the video frames more objectively and accurately, and the accuracy and stability of subsequent identification using the obtained target video frames are also higher.

[0144] Based on this application Figure 2 Corresponding embodiments, in this embodiment, the same or similar contents as those in the above embodiments can be referred to the above description, and will not be described in detail later. On this basis, a specific implementation method for scoring the image quality of video frames is provided, please refer to Figure 3 , Figure 3 The third flow chart of the method for identifying a person in a vehicle compartment provided by the present application, step S201 includes steps S2011 to S2012:

[0145] Step S2011, performing the following steps on each of the N video frames:

[0146] 1) evaluating the image clarity of the video frame to obtain a first score;

[0147] In some embodiments, a specific implementation method for evaluating the first score is provided. The specific implementation method for evaluating the image clarity of the video frame to obtain the first score may include the following steps:

[0148] Step 1-1, using the Sobel edge detection method to calculate the gradient of the pixel grayscale value in the video frame; wherein the gradient of the pixel grayscale value in the video frame is used to characterize the image clarity;

[0149] Step 1-2, obtaining a first candidate score for the video frame based on the mean value of the pixel grayscale value gradient in the video frame;

[0150] It should be noted that the Sobel edge detection method can detect edges in video frames. The more edges there are and the clearer they are, the higher the image clarity is. In the above steps, the larger the mean value of the pixel grayscale gradient in the video frame is, the higher the image clarity is. It is simple to implement, with relatively small amount of calculation, and can detect details in the image well. However, it is sensitive to noise, which may lead to erroneous edge detection, so it is necessary to detect edges of different scales to improve robustness.

[0151] Step 1-3, using the Laplacian gradient detection method to calculate the second-order derivative of the pixel grayscale value in the video frame; wherein the second-order derivative of the pixel grayscale value in the video frame is used to characterize the image clarity;

[0152] Step 1-4, obtaining a second candidate score for the video frame by calculating the Laplace gradient variance based on the second-order derivative of the pixel grayscale value in the video frame;

[0153] It should be noted that in the Laplacian gradient detection method, operators such as the Laplacian variance can be used to calculate the second-order derivative of the pixel grayscale value in the video frame to characterize the image clarity. The amount of calculation is relatively small, but the gradient detection method may not be able to fully reflect the overall clarity of the image.

[0154] Step 1-5: Determine a first score based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight.

[0155] In summary, the present application adopts the method of fusing Sobel edge detection and Laplacian gradient detection to evaluate image clarity, which can effectively reduce the interference of noise on the clarity of video frame images, thereby improving the robustness of detecting image clarity.

[0156] It should be noted that the above steps 1-1 and 1-2 are used to evaluate the first candidate score, and the above steps 1-3 and 1-4 are used to evaluate the second candidate score. The present application does not limit the order between evaluating the first candidate score and evaluating the second candidate score. The above step diagrams are only for illustration and the present application does not impose any restrictions on this.

[0157] Specifically, the image clarity value Q of video frame i can be calculated by combining the Sobel edge detection method and the Laplacian gradient detection method: i , which can be calculated using the following formula:

[0158] Q i =α*Q Sobel +β*Q Laplacian ;

[0159] Among them, Q Sobel represents the first candidate score calculated by the Sobel edge detection method, Q Laplacian represents the second candidate score calculated by the Laplacian gradient detection method, where Q Sobel and Q Laplacian The value range of has been normalized to 0-100, α represents the fourth weight corresponding to the first candidate score, and β represents the fifth weight corresponding to the second candidate score. They can be adjusted according to experimental results and scenarios, for example, they can be set to 0.5.

[0160] In some embodiments, a deep learning-based method can also be used to calculate the image clarity of video frames. For example, by training a deep learning model such as CNN (Convolutional Neural Network) to evaluate image clarity, the model can learn more complex features, thereby effectively improving the accuracy of the score.

[0161] In this embodiment, when evaluating the first score, the present application adopts a fusion method of Sobel edge detection and Laplacian gradient detection to evaluate image clarity, which can effectively reduce the interference of noise on the clarity of the video frame image, thereby improving the robustness of the detected image clarity, and is conducive to more accurate and objective evaluation of the first score, and selecting a target video frame that is more helpful in evaluating whether the truck bed is carrying people, thereby improving the accuracy and stability of subsequent identification of whether there are people in the truck bed.

[0162] In some embodiments, a specific implementation of determining the first score is provided. The above steps 1-5 may include the following sub-steps:

[0163] Sub-step 1-51, obtaining a third candidate score for the video frame based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight;

[0164] It should be noted that the third candidate score here is the Q i .

[0165] Sub-step 1-52, performing K-Medoids clustering on the third candidate scores corresponding to the N video frames respectively to obtain K clusters; wherein the difference between the third candidate scores corresponding to the video frames in the same cluster is less than the first value; wherein K represents the number of categories, and K is an integer greater than 1;

[0166] Sub-step 1-53, sorting the K clusters in ascending order according to the central value of the third candidate score in each cluster to obtain ordered K clusters;

[0167] Sub-step 1-54, for each of the ordered K clusters, based on the sequence number of the cluster in the ordered K clusters and K, determine the clarity score of the cluster as the first score corresponding to all video frames in the cluster.

[0168] Specifically, the third candidate score Q corresponding to all video frames in N video frames is i The values ​​are clustered by K-Medoids to obtain K clusters with K-class distribution, where K>1. For example, the number of categories K can be set to 3, that is, according to Q i The N video frames are divided into three clusters, each of which corresponds to at least one video frame and its corresponding Q i value.

[0169] It should be noted that K can be set according to actual conditions, and this application does not impose any restrictions on this.

[0170] Then divide the K clusters into clusters according to each cluster Q i The central values ​​of the values ​​are sorted from small to large to obtain K ordered clusters C k (1≤k≤K), k represents the kth cluster (it can also be considered as the serial number of different clusters in the ordered K clusters). As the value of k increases, C k The larger the center value.

[0171] After sorting the K clusters, the final clarity score of the video frame i in each cluster can be mapped by the following formula, that is, the first score Score_Q i ':

[0172]

[0173] In this embodiment, K-Medoids uses actual points as clustering centers for clustering, that is, it uses the third candidate score instead of the virtual value for clustering, thereby achieving stronger robustness against noise and outliers, and can identify outliers (i.e., values ​​with large fluctuations) and filter them; in addition, mapping K clusters can more evenly distribute the clarity and improve the stability of the determined first score.

[0174] 2) evaluating the completeness of the target vehicle in the video frame to obtain a second score;

[0175] In some embodiments, a specific implementation method for evaluating the second score is provided. The specific implementation method for evaluating the completeness of the target vehicle in the video frame to obtain the second score may include the following steps:

[0176] Step 2-1, inputting the video frame into a pre-trained target detection model, and using the target detection model to evaluate the integrity of the target vehicle in the video frame to obtain a second score;

[0177] The training set used when training the target detection model includes samples with occluded targets and samples with incomplete targets.

[0178] It should be noted that the target detection model is implemented by, for example, a YOLO (You Only Look Once) training model.

[0179] Specifically, for the evaluation of the completeness of the target vehicle in the video frame, the improved YOLO target detection model can be used for evaluation:

[0180] <1> Integrity assessment based on IOU (Intersection over Union);

[0181] Based on the target integrity evaluation scenario, during YOLO training, samples with occluded and incomplete targets can be added to the training set, and an IOU-based integrity loss evaluation can be added to obtain target integrity prediction results and improve detection accuracy.

[0182] <2> Completeness score quantification;

[0183] Since the IOU value predicted by the YOLO model is in the range of 0-1 and directly reflects the completeness of the target vehicle in the video frame, for the convenience of representation and comparison, we directly multiply the IOU value (also known as IOU_Value) by 100 and convert it to a score range of 0-100:

[0184] Score_CPT=IOU_Value*100.

[0185] It should be noted that using the YOLO model to evaluate the integrity of the target vehicle in the video frame has at least the following beneficial effects:

[0186] A. Improve evaluation accuracy: By adding IOU-based integrity loss evaluation during YOLO training, the model can better learn the integrity characteristics of the target vehicle, thereby significantly improving detection accuracy. In related technologies, traditional bounding box-based evaluation methods usually rely on fixed thresholds to judge the integrity of the target vehicle. This method is prone to false detection and missed detection in the case of complex backgrounds or partial occlusion of the target. Compared with the above method, this application introduces IOU-based integrity loss evaluation, which can more accurately judge the integrity of the target vehicle, reduce false detection and missed detection, and more accurately detect and locate the target in complex backgrounds and occlusions.

[0187] B. Reduce false detection and missed detection: IOU-based integrity loss assessment can effectively reduce false detection and missed detection, and can effectively evaluate the integrity of the target vehicle in the video frame.

[0188] C. Simplified scoring quantification: Traditional scoring methods may require complex calculation and conversion steps. This application simplifies the quantification process of completeness scoring by directly multiplying the IOU value by 100, making the scoring results more intuitive and easy to compare, while reducing the calculation complexity.

[0189] In this embodiment, when evaluating the second score, the present application uses a pre-trained target detection model to evaluate the integrity of the target vehicle in the video frame to obtain the second score, which is conducive to a more objective, accurate and rapid integrity scoring. In addition, samples with occluded targets and incomplete targets can be added to assist in training the target detection model, which helps the model to more accurately evaluate the integrity of the target vehicle in the video frame and score it, thereby meeting the accuracy and real-time requirements of vehicle bucket recognition.

[0190] 3) evaluating the prominence of the target vehicle in the video frame to obtain a third score;

[0191] In some embodiments, a specific implementation method for evaluating the third score is provided. The specific implementation method for evaluating the prominence of the target vehicle in the video frame to obtain the third score may include the following steps:

[0192] Step 3-1, inputting the video frame into a preset saliency recognition model, wherein the saliency recognition model evaluates the saliency of the target area in the video frame, and outputs the grayscale value corresponding to each pixel in the target area; wherein the target area is the area of ​​the target vehicle in the video frame, and the grayscale value corresponding to each pixel in the target area represents the saliency strength of each pixel in the video frame;

[0193] It should be noted that the above saliency recognition model is implemented by using, for example, the ITTI model.

[0194] Step 3-2: Determine a third score based on the sum of the grayscale values ​​corresponding to all pixels in the target area.

[0195] It should be noted that the third score is used to characterize the prominence of the target vehicle in the video frame, and can also be used to evaluate the prominence of the truck bed and the people therein in the video frame. Prominence detection can simulate the human visual attention mechanism and identify the most eye-catching areas in the video frame.

[0196] Specifically, for the evaluation of the prominence of the target vehicle in the video frame, the following steps can be adopted:

[0197] <1> Saliency map generation;

[0198] The ITTI model is used to process the frame area containing the truck and its personnel (i.e., the target area mentioned above) to generate the corresponding saliency map S. The saliency map S is a grayscale image of the same size as the original video frame, in which the grayscale value represents the saliency intensity of different areas in the video frame.

[0199] <2> Gray value normalization;

[0200] The sum S' of the grayscale values ​​of the target area in the above saliency map S is normalized to the range of 0-1 to eliminate the problem that the grayscale value range may be inconsistent between different video frames:

[0201]

[0202] Wherein, Pn is the number of pixels in the target area, and Smax is the maximum grayscale value in S. For example, for an 8-bit image, Smax is 255.

[0203] (3) Scoring quantification;

[0204] For the convenience of representation and comparison, the grayscale value of the normalized saliency map Snorm is linearly mapped to 0-100: Score_SIG = Snorm*100.

[0205] It should be noted that using the ITTI model to evaluate the prominence of the target vehicle in the video frame has at least the following beneficial effects:

[0206] A. Multi-scale and multi-feature fusion: Traditional methods based on a single scale or a single feature (such as color-based saliency detection) can often only capture partial saliency information and easily miss some important details; however, this application uses the ITTI model to capture saliency information of different scales and different features through multi-scale and multi-feature fusion. This multi-scale and multi-feature fusion enables the ITTI model to more comprehensively evaluate the saliency of the target vehicle in the video frame.

[0207] B. Robustness: Traditional methods based on fixed thresholds or single features are prone to false detection and missed detection in complex environments. However, this application uses the ITTI model to better handle environmental factors such as lighting changes and background complexity through the fusion of multi-scale and multi-features, thereby improving the robustness of the system. This robustness enables the ITTI model to maintain a high detection accuracy in various complex environments.

[0208] C. Computational efficiency: Although some saliency detection methods based on deep learning have improved in accuracy, their computational complexity is high and they are difficult to use in real-time applications. Although the ITTI model used in this application involves the fusion of multiple scales and multiple features, its computational efficiency is relatively high. Through reasonable algorithm design and optimization, the ITTI model can maintain high computational efficiency in real-time applications and ensure the real-time performance of scoring and subsequent recognition.

[0209] In this embodiment, when evaluating the third score, the present application adopts a pre-set saliency recognition model to evaluate the saliency of the target vehicle in the video frame, and uses the grayscale value of the pixel point as the output to characterize the saliency intensity of the pixel point in the video frame, and then obtains the third score based on the sum of all grayscale values ​​in the target area where the target vehicle is located, which is conducive to more objective, accurate and rapid saliency scoring, and meets the accuracy and real-time requirements of vehicle bucket identification.

[0210] Step S2012: Calculate the target quality comprehensive score corresponding to the video frame based on at least two of the first score and the corresponding first weight, the second score and the corresponding second weight, and the third score and the corresponding third weight.

[0211] Specifically, after evaluating and obtaining the first score, the second score, and the third score, the target quality comprehensive score corresponding to the video frame can be comprehensively calculated:

[0212] First, it is necessary to assign a weight to each dimension based on a large amount of test data from actual scenarios according to the importance of each dimension, that is, to determine the first weight corresponding to the first score, the second weight corresponding to the second score, and the third weight corresponding to the third score.

[0213] For each video frame i, the score of each dimension is multiplied by its corresponding weight, and then summed up to obtain the final target quality comprehensive score Quality(i):

[0214] Quality(i)=(Score_Q i '*first weight)+(Score_CPT*second weight)+(Score_SIG*third weight).

[0215] In an embodiment of the present application, when scoring video frames, the present application considers several objective aspects such as image clarity, completeness and prominence of the target vehicle in the video frame for scoring, and obtains the comprehensive target quality score corresponding to the video frame through weighted summation, and then determines the final scoring result based on the score. For example, the higher the image clarity, the higher the completeness, and the higher the prominence of the video frame, the higher the score obtained. Subsequent recognition using video frames with higher scores is more helpful in accurately identifying the presence of people in the truck bed, thereby improving the accuracy and stability of identifying people in the truck bed.

[0216] Based on this application Figure 2 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be repeated in the following. On this basis, a specific implementation method for scoring dynamic information of video frames is provided, please refer to Figure 4 , Figure 4 The fourth flow chart of the method for identifying a person in a vehicle compartment provided by the present application, step S202 includes steps S2021 to S2024:

[0217] The following steps are performed for each of the N video frames:

[0218] Step S2021, extracting key dynamic information corresponding to the video frame;

[0219] The key dynamic information includes at least one of optical flow information and feature point motion information, wherein the optical flow information represents the motion of the feature points in the video frame between two consecutive frames, and the feature point motion information represents the position change of the feature points in the video frame in the time series;

[0220] Step S2022, calculating relevant parameters of feature points in the video frame based on the key dynamic information;

[0221] The relevant parameters include at least one of a motion intensity change rate, a complex intensity of a moving path, a speed change, and an acceleration change. The motion intensity change rate represents a change rate between a total displacement of a feature point in the video frame or a modulus of an optical flow vector and a previous video frame. The complex intensity of the moving path represents a ratio of a moving path distance of the feature point in the video frame to a straight-line distance between a starting point and an end point of the moving path. The speed change represents a speed change of a feature point in the video frame between a total of m video frames before and after the video frame. The acceleration change represents a acceleration change of a feature point in the video frame between a total of m video frames before and after the video frame, where m is an integer greater than 2.

[0222] Step S2023, determining a candidate score of dynamic information corresponding to the video frame based on relevant parameters of the feature points in the video frame;

[0223] Step S2024: Determine the dynamic information score of the video frame based on the dynamic information candidate scores corresponding to the video frame and the previous video frame respectively.

[0224] Specifically, in the task of identifying passengers in a truck bed, accurately evaluating the quality and value of the dynamic features of video frames is crucial to improving recognition accuracy. The purpose of this application is to select video frames with rich target dynamic information to provide high-quality data for subsequent recognition.

[0225] For dynamic information scoring, the following steps can be performed:

[0226] <1> Feature extraction: Extract key dynamic information from each video frame, including but not limited to:

[0227] a. Optical flow information: reflects the movement of pixels or feature points between two consecutive frames.

[0228] b. Feature point motion information: reflects the position changes of feature points obtained by the feature point tracking algorithm in the time series.

[0229] <2> Feature calculation: Based on the extracted key dynamic information, calculate the feature values ​​of each video frame as the relevant parameters of the feature points, including but not limited to:

[0230] a. Motion intensity change rate: Calculate the total displacement of the feature point movement or the modulus of the optical flow vector in each video frame, and compare it with the previous video frame to obtain the change rate. The larger the change rate, the more significant the change in the motion intensity of the video frame, which may contain rich dynamic information.

[0231] b. Complexity intensity of the moving path: Analyze the moving path of the feature point and calculate the ratio of the path length to the straight-line distance from the starting point to the end point. The higher the complexity intensity of the path, the more complex the feature point has experienced in the video frame.

[0232] c. Speed ​​change and acceleration change: Calculate the speed change and acceleration change of feature points between consecutive frames. Significant speed or acceleration changes may mean the presence of important dynamic events.

[0233] <3> Dynamic information scoring: Assign weights to each feature based on its importance and calculate the comprehensive score for each video frame. The weight assignment can be determined based on actual application scenarios and expert experience:

[0234] DFScore i =ω1*rate of change of motion intensity+ω2*complex intensity of moving path+ω3*speed change+ω4*acceleration change;

[0235] Among them, ω1 is the weight corresponding to the rate of change of motion intensity, ω2 is the weight corresponding to the complex intensity of the moving path, ω3 is the weight corresponding to the change in speed, and ω4 is the weight corresponding to the change in acceleration.

[0236] In some embodiments, after the target quality comprehensive score and the dynamic information score are calculated, dynamic multi-frame optimization can be performed to screen out the target video frames, with the goal of selecting multiple target video frames with the largest sum of target quality scores and inter-frame differences among the N video frames from the appearance to the disappearance of the target vehicle. Figure 5 For a schematic diagram of the target from appearing to disappearing in the vehicle load identification method provided by this application, please refer to Figure 5 In the figure, id1 is considered to be the target vehicle, and F1, F2, F3...FN are considered to be N video frames. Specifically, the following steps can be performed:

[0237] 1) Define quality score: Based on the above score output, each video frame has a corresponding target quality comprehensive score Quality(i), where i represents the sequence number of the video frame (i ranges from 2 to N).

[0238] 2) Define inter-frame differences: For any adjacent frames, their difference values ​​can be calculated, denoted as Diff(i), which represents the difference between the i-th frame and the i-1-th frame. In order to measure the inter-frame differences, this application calculates Diff(i) based on the dynamic information score difference between adjacent frames:

[0239]

[0240] Among them, DFScore i is the dynamic information score of the i-th frame, DFScorei-1 is the dynamic information score of the i-1th frame, DFScore max It is the maximum value of the difference in dynamic information scores between adjacent frames.

[0241] The identification of passengers in a truck bed usually relies on information in a video or image sequence. The greater the difference between frames, the richer the dynamic information between frames, thus avoiding misidentification due to noise, occlusion or misjudgment in a single frame image.

[0242] 3) Taking into account the target quality comprehensive score and the difference between frames, this application designs an optimization function F(Q,D) to select the optimal frame (that is, the target video frame):

[0243]

[0244] Among them, σ+λ=1, σ is the weight coefficient corresponding to the target quality comprehensive score, and λ is the weight coefficient corresponding to the inter-frame difference, which can be set according to actual conditions.

[0245] 4) Select multiple video frames with larger optimal function values ​​F(Q, D, i), for example, select three frames with larger F(Q, D, i), for subsequent analysis and recognition.

[0246] In an embodiment of the present application, in the scoring stage, dynamic information scoring of video frames can also be considered to select video frames with rich target dynamic information to provide high-quality data for subsequent identification. Finally, based on the comprehensive target quality score and dynamic information score corresponding to the video frame, the scoring results corresponding to the video frame are jointly determined to perform video frame screening, which is conducive to more effective subsequent identification of people in the vehicle bed.

[0247] Based on this application Figure 1 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be described in detail later. Figure 6 , Figure 6 The fifth flow chart of the method for identifying a person in a vehicle compartment provided by the present application may further include step S50 before step S20:

[0248] Step S50, for each of the N video frames, performing the following steps: detecting and selecting a framed area where the target vehicle is located; and expanding the framed area according to a preset scaling ratio as a target detection area of ​​the video frame;

[0249] Wherein, the scaling ratio is greater than 1.

[0250] It should be noted that the zoom ratio can be set according to actual conditions, and the zoom ratio needs to be greater than 1 so that the target detection area includes a portion of the surrounding environment area in addition to the framed area. For example, the zoom ratio can be set to 1.2.

[0251] Correspondingly, step S20 needs to be replaced by step S20-1:

[0252] Step S20-1, scoring the target detection area of ​​each of the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result.

[0253] Specifically, the above scoring process can be performed in the target detection area that is r times larger than the framed area where the target vehicle is located, where r is the scaling ratio, and the value range is greater than 1, and is set to 1.2 here, for example. Compared with the full-image calculation, the amount of calculation is greatly reduced; compared with the analysis of the original framed area, the r-fold expansion provides richer background information for subsequent analysis.

[0254] In an embodiment of the present application, compared to calculating and scoring the entire video frame, the present application selects the target vehicle and the surrounding area as the target detection area, which greatly reduces the amount of calculation; in addition, compared to only analyzing the selected area where the target vehicle is located, the selected area is expanded to the target detection area including the surrounding area by scaling the ratio for subsequent scoring, which can provide richer background information for subsequent analysis, and is conducive to more accurate scoring and screening of multiple target frames with the largest amount of information.

[0255] Based on this application Figure 1 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, the recognition model may include a vehicle type recognition module, a truck bed load recognition module, and a truck bed passenger recognition module, and the recognition result may include a first result output by the vehicle type recognition module, a second result output by the truck bed load recognition module, and a third result output by the truck bed passenger recognition module, please refer to Figure 7 , Figure 7 The sixth flow chart of the method for identifying a person in a vehicle compartment provided by the present application, step S30 may include step S301:

[0256] Step S301: for each of the at least two target video frames, perform the following steps:

[0257] 1) inputting the target video frame into the vehicle type recognition module, and the vehicle type recognition module recognizes the vehicle type of the target vehicle from the target video frame as the first result;

[0258] It should be noted that vehicle types include but are not limited to large trucks, small trucks, pickup trucks, buses, sedans, vans, and SUVs (Sport Utility Vehicle). Among them, large trucks, small trucks, or pickup trucks are considered to be equipped with a cargo box.

[0259] 2) inputting the target video frame into the bucket loading recognition module, and the bucket loading recognition module recognizes the bucket loading state of the target vehicle from the target video frame as the second result;

[0260] Wherein, the loading state of the truck bucket includes that the truck bucket is loaded or the truck bucket is not loaded;

[0261] 3) inputting the target video frame into the vehicle load passenger recognition module, and the vehicle load passenger recognition module recognizes the vehicle load passenger status of the target vehicle from the target video frame as the third result;

[0262] The loading state of the vehicle bucket includes the loading state of the vehicle bucket or the unloading state of the vehicle bucket;

[0263] Among them, the vehicle type recognition module, the truck bed loading recognition module and the truck bed passenger recognition module are trained based on a multi-task learning method.

[0264] It should be noted that the multi-task learning method can capture the correlation between different tasks by sharing the underlying features, thereby improving the performance of each task. This is mainly because in the identification of truck loads, there is a certain correlation between the vehicle model, truck load status and truck load status. For example, when the truck loads people, the vehicle model should be a large truck, a small truck or a pickup truck, and the truck load status should be non-empty. Therefore, multi-task learning can make full use of these correlations to improve the recognition accuracy.

[0265] It should also be noted that the above recognition model contains three task branches: vehicle type recognition, truck loading status recognition, and truck passenger status recognition. Each task branch uses a different network structure or algorithm to adapt to the characteristics of different tasks:

[0266] <1> Vehicle type recognition task: usually involves the recognition of the overall or local features of the vehicle, which may require a high level of detail capture capability.

[0267] For the vehicle type recognition task, a deep convolutional neural network was designed as the backbone network for vehicle type classification, which was then connected to multiple convolutional layers and combined with a softmax classifier to classify different vehicle types. The vehicle types are classified into seven categories: large trucks, small trucks, pickup trucks, buses, sedans, vans, and SUVs.

[0268] <2> Truck bed loading status recognition: Pay attention to the changes in the vehicle loading status, and may focus more on the characteristics of the truck bed area.

[0269] For the identification of the loading state of the truck bucket, a recognition network that focuses more on the truck bucket area is constructed. The ROI Align (Region of Interest Align) technology is used to accurately extract the high-resolution features of the truck bucket area from the shared feature map, and then the loading state of the truck bucket is determined by the classifier. The loading state of the truck bucket is divided into two types: empty (that is, the truck bucket is not loaded) and non-empty (that is, the truck bucket is loaded).

[0270] <3> Identification of passenger status in the truck bed: It is necessary to identify whether there is anyone in the truck bed, and sensitive capture of human body features may be required.

[0271] For the identification of people in the truck bed, in addition to the characteristics of the truck bed area, we also need to pay attention to the characteristics of the human body. Therefore, after the shared feature layer, a human detection subnetwork based on the YOLO algorithm can be embedded. This network can efficiently identify human targets in the image and combine the analysis results of the truck bed area to comprehensively judge whether there are people in the truck bed. The results of people in the truck bed are divided into two types: carrying people (that is, the truck bed mentioned above has people on it) and not carrying people (that is, the truck bed mentioned above has no people on it).

[0272] In the embodiment of the present application, the recognition model is a model based on multi-task learning. In the identification of cargo bed passengers, there is a certain correlation between the vehicle model, cargo bed loading status and cargo bed loading status. For example, when the cargo bed is loaded with people, the vehicle model should be a large truck, a small truck or a pickup truck type with a cargo bed, and the cargo bed loading status should be non-empty. The present application adopts multi-task learning to make full use of these related information and effectively improve the recognition accuracy.

[0273] In some embodiments, a specific implementation method for building and training a recognition model is provided. Before the above step S301 or step S30, the following steps may also be performed:

[0274] Step 4-1, using a pre-trained shared feature extractor to extract shared features in the training set, and constructing a shared backbone network based on the shared features;

[0275] Step 4-2, using the training set to respectively train a preset first branch network, a second branch network, and a third branch network;

[0276] It should be noted that the training sets used by the first branch network, the second branch network and the third branch network are different. The training set used by the first branch network is related to vehicle type recognition, the training set used by the second branch network is related to truck loading status recognition, and the training set used by the third branch network is related to truck loading status recognition. Each branch network learns its own unique parameter set, so as to flexibly respond to the differences between tasks.

[0277] Step 4-3: construct a vehicle type recognition candidate network based on the shared backbone network and the trained first branch network, construct a vehicle loading recognition candidate network based on the shared backbone network and the trained second branch network, and construct a vehicle loading candidate network based on the shared backbone network and the trained third branch network;

[0278] Step 4-4, using a preset multi-task loss function, respectively train the vehicle type recognition candidate network, the truck bed loading recognition candidate network and the truck bed passenger candidate network to obtain the vehicle type recognition module, the truck bed loading recognition module and the truck bed passenger recognition module; wherein the multi-task loss function is the weighted sum of the loss functions of the vehicle type recognition candidate network, the truck bed loading recognition candidate network and the truck bed passenger candidate network.

[0279] Specifically, Figure 8 For a schematic diagram of multi-task learning in a deep neural network in the method for identifying passengers in a truck bed provided in this application, please refer to Figure 8 At the bottom layer of the model, ResNet-50 can be used as a shared feature extractor to output shared feature maps, providing high-level semantic input for the three branch tasks, effectively capturing the multi-level and multi-scale features in the video frames. This design not only enhances the expressiveness of features, but also promotes information sharing between different tasks. In addition, a soft parameter sharing strategy is also adopted to allow each task branch to learn its own unique parameter set based on the shared underlying features (through the above-mentioned shared backbone network), so as to flexibly deal with the differences and correlations between tasks, and further improve the generalization ability and performance of the overall model.

[0280] Specifically, a large amount of vehicle data is annotated through public and self-owned data sets as training sets for training, and data enhancement methods such as image flipping, cropping, and occlusion are used to expand the training set to enhance the generalization ability of the model. The adamW optimizer can be used in training, with an initial learning rate of 0.001 and a weight decay of 0.01. The specific training steps are as follows:

[0281] 1) Pre-training the shared feature extractor: The shared feature extractor is pre-trained on a large-scale vehicle image training set.

[0282] 2) Branch training: Each task branch is first trained separately and specific parameters are initialized;

[0283] 3) Joint training: Use a multi-task loss function to jointly optimize shared features and branch features. Specifically, the multi-task loss function L total It is obtained by weighted combination of the loss functions of the three tasks:

[0284]

[0285] Among them, L1 is the loss function of the above model recognition candidate network, is the weight coefficient corresponding to L1, L2 is the loss function of the above-mentioned truck loading recognition candidate network, is the weight coefficient corresponding to L2, L3 is the loss function of the above-mentioned candidate network for loading passengers. is the weight coefficient corresponding to L3. Each weight coefficient can be set according to the actual situation. For example, all weight coefficients are set to 1, which means that each task is equally important.

[0286] Based on this application Figure 7 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be described in detail later. Fig. 9 , Fig. 9 The seventh flow chart of the method for identifying a person in a vehicle compartment provided by the present application, in step S40, may include steps S401 to S404:

[0287] Step S401: for each of the at least two target video frames, determine whether a first result corresponding to the target video frame is a target vehicle type with a truck bed.

[0288] Step S402, when it is determined that the first result corresponding to the target video frame is the target vehicle type, it is determined whether the second result corresponding to the target video frame is that the truck box is loaded.

[0289] Step S403, when it is determined that the second result corresponding to the target video frame is that the bucket is loaded, the third result corresponding to the target video frame is taken as a valid result.

[0290] Step S404: Based on the valid results corresponding to the at least two target video frames, determine whether the vehicle compartment of the target vehicle is carrying people.

[0291] Specifically, the first result, the second result and the third result of the identification can be first processed through logical verification. Specifically, when the vehicle type identification result of each video frame (that is, the first result) is one of a large truck, a small truck or a pickup truck, and the loading status of the cargo box (that is, the second result) is that the cargo box is loaded, then attention will be paid to the cargo box passenger identification result of each video frame (that is, the third result). At this time, the third result is considered to be a valid result and further judgment is performed; otherwise, it is considered that the cargo box is unloaded. That is, when the first result corresponding to the target video frame is determined not to be the target vehicle type, or the second result corresponding to the target video frame is determined to be that the cargo box is unloaded, the third result corresponding to the target video frame is directly determined as an invalid result, and it is determined that the cargo box is unloaded.

[0292] In an embodiment of the present application, in the result fusion stage, the recognition results of multiple recognition modules can be verified by means of logical verification. Specifically, it is first determined whether the first result corresponding to the target video frame is the target vehicle type with a cargo box. If so, it is further determined whether the second result is in a state where the cargo box is loaded. If so, it is considered that the target vehicle in the target video frame may have people in the cargo box, and the corresponding third result can be used as a valid result for subsequent processing. If the first result corresponding to the target video frame is not the target vehicle type with a cargo box, then there is definitely no situation where there are people in the cargo box. Similarly, if the cargo box loading state of the vehicle with a cargo box is that the cargo box is unloaded, then there is definitely no situation where there are people in the cargo box. At this time, there is no need to consider the third result. By means of logical verification, the computational complexity of the recognition process can be effectively reduced, and the efficiency of recognizing people in the cargo box can be improved.

[0293] In some embodiments, the above step S404 may include the following steps:

[0294] Step 5-1. When it is determined that the number of corresponding valid results in the at least two target video frames is an odd number, compare the first number of valid results indicating that the vehicle bed is loaded with people with the second number of valid results indicating that the vehicle bed is not loaded with people. If the first number is greater than the second number, it is determined that the vehicle bed of the target vehicle is loaded with people; if the first number is less than the second number, it is determined that the vehicle bed of the target vehicle is not loaded with people.

[0295] Step 5-2: When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence among the valid results is greater than or equal to the second value, if the valid result with the highest confidence indicates that the cargo box is loaded with people, then it is determined that the cargo box of the target vehicle is loaded with people; if the valid result with the highest confidence indicates that the cargo box is not loaded with people, then it is determined that the cargo box of the target vehicle is not loaded with people.

[0296] Step 5-3: When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence level among the valid results is less than a second value, it is determined that the cargo box of the target vehicle is unmanned.

[0297] In the embodiment of the present application, the voting method and the confidence ranking method are used to merge the results according to whether the number of valid results obtained above is an odd number or an even number. Specifically, when there are an odd number of valid results, if there are more valid results indicating that the bucket is loaded, it is determined that the bucket is loaded, otherwise it is determined that the bucket is not loaded; when there are an even number of valid results, a tie may occur through the voting method, so the present application sorts the valid results according to their confidence, and determines whether the bucket of the target vehicle is loaded based on the valid result with the highest confidence.

[0298] For the confidence ranking method, it can be divided into the following two cases:

[0299] Case 1: When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence among the valid results is greater than or equal to the second value, if the valid result with the highest confidence indicates that the cargo box is loaded with people, then it is determined that the cargo box of the target vehicle is loaded with people; if the valid result with the highest confidence indicates that the cargo box is not loaded with people, then it is determined that the cargo box of the target vehicle is not loaded with people.

[0300] Case 2: When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence level among the valid results is less than a second value, it is determined that the cargo box of the target vehicle is unmanned.

[0301] For example, the above three highest quality snapshots are selected as target frames through scoring, and three corresponding vehicle load recognition results (i.e., the third result) and their confidence levels can be obtained. In order to improve the robustness and accuracy of the overall recognition, this application uses a voting method combined with a confidence ranking method to fuse the results. Specifically:

[0302] 1) Voting method: The valid result of each video frame is regarded as a vote, the valid votes for each valid result are counted, and the valid result with the most votes is taken as the final output.

[0303] 2) Confidence sorting method: If there are two valid results, the confidence sorting method is used to determine the final result of the loading of passengers in the truck bed, and the valid result with the highest confidence is selected as the final result output.

[0304] It should be noted that if the confidence level is lower than 0.5, the result is considered invalid. If all results are invalid, it is determined that there is no one in the cargo box of the target vehicle.

[0305] It should also be noted that the above-mentioned second value can be set according to actual conditions. The larger the second value is set, the more accurate the reporting of the passenger situation in the vehicle bed will be.

[0306] In the embodiment of the present application, if the confidence level is lower than the second value, the result is considered invalid. If all results are invalid, it is determined that the cargo box of the target vehicle is unloaded, which can ensure accurate reporting of the cargo box loading situation.

[0307] The following example illustrates the method for identifying passengers in a vehicle bed provided in an embodiment of the present application.

[0308] The method for identifying passengers in the cargo compartment can be implemented through the following three modules: optimal frame capture module, cargo compartment passenger recognition module based on multi-task learning and result fusion module;

[0309] Fig.10 Please refer to the flowchart of the method for identifying passengers in the vehicle compartment provided by this application. Fig.10 First, the video stream collected by the acquisition device is obtained, and then the optimal frame capture module detects, tracks the target vehicle, scores and selects frames for multiple video frames in the video stream in turn, and outputs the selected three target video frames P1, P2, and P3; then the cargo bed passenger recognition module based on multi-task learning performs vehicle model recognition, cargo bed loading status recognition and cargo bed passenger recognition on each target video frame, and obtains P1 result, P2 result, and P3 result; then the result fusion module fuses the P1 result, P2 result, and P3 result through logical verification, voting method, and confidence ranking method to determine whether the cargo bed is loaded with people.

[0310] The following is a detailed description of the process of each module:

[0311] 1) Optimal frame capture module;

[0312] In order to capture a clear target vehicle and ensure the accuracy of subsequent analysis results, this application proposes to capture the first three frames with the highest target quality based on the video stream as the input of the subsequent recognition model, which can make up for the inaccurate recognition of a single frame caused by occlusion, blur, etc.

[0313] Fig.11 Please refer to the flowchart of the optimal frame capture module in the method for identifying passengers in the vehicle compartment provided by this application. Fig.11 , including the following steps:

[0314] Step 6-1, obtain the video stream;

[0315] Step 6-2: Target detection and tracking of video frames;

[0316] Step 6-3, video frame scoring;

[0317] Step 6-4, dynamic multi-frame optimization;

[0318] Step 6-5: Get the best three frames as target video frames.

[0319] For the system, the system obtains the video stream in real time through the camera. The detection module can detect the target vehicle in the video frame, and then the tracking module uniquely identifies each target vehicle. The scoring module performs multi-dimensional scoring for each video frame to determine the quality of the target vehicle in the video frame.

[0320] <1> Target frame scoring module;

[0321] The quality of the target vehicle in the current frame is evaluated and quantified based on the following dimensions to provide the accuracy of the target quality comprehensive score:

[0322] For image quality rating:

[0323] A. Image clarity: Dynamic inter-frame clarity assessment based on edge and gradient detection.

[0324] B. Target completeness: Use the improved YOLO target detection algorithm to evaluate the completeness of the target vehicle in the video frame.

[0325] C. Target saliency: Evaluate the saliency of the truck and its personnel in the video frame. The saliency detection algorithm can simulate the human visual attention mechanism and identify the most eye-catching areas in the video frame.

[0326] In the comprehensive scoring stage, a weight is assigned to each dimension based on the importance of each of the above dimensions, and the weighted sum is used to obtain the comprehensive score of the target quality.

[0327] For dynamic information scoring: In the task of identifying passengers in the truck bed, accurately evaluating the quality and value of dynamic features is crucial to improving recognition accuracy. This application selects frames with rich target dynamic information to provide high-quality data for subsequent recognition.

[0328] <2> Dynamic multi-frame optimization module: The purpose is to select the three frames with the largest sum of target quality comprehensive scores and frame differences among the N frames from the appearance to the disappearance of the target vehicle as the target video frames. The specific implementation refers to the above embodiment and will not be repeated here.

[0329] 2) A vehicle load identification module based on multi-task learning;

[0330] Multi-task learning can capture the correlation between different tasks by sharing underlying features, thereby improving the performance of each task. In the task of identifying people in the cargo bed, there is a certain correlation between the vehicle model, the loading status of the cargo bed, and the number of people in the cargo bed. For example, when the cargo bed is loaded with people, the vehicle model should be a large truck, a small truck, or a pickup truck, and the loading status of the cargo bed should not be empty. Therefore, multi-task learning can make full use of these correlations to improve the recognition accuracy. For specific implementation, please refer to the above embodiment.

[0331] 3) Result fusion module;

[0332] Fig.12 Please refer to the flowchart of the result fusion module in the method for identifying passengers in the vehicle compartment provided by this application. Fig.12 , first perform a logical check. When the vehicle type recognition result of each video frame is a large truck, a small truck or a pickup truck, and the loading status of the cargo box is not empty, then pay attention to the recognition results of the cargo box carrying people in the three target video frames. Otherwise, it is considered to be unloaded.

[0333] Since the above optimal frame capture module obtains three highest quality capture images as three target video frames, three corresponding human recognition results and their confidences can be obtained. In order to improve the overall robustness and accuracy, this application adopts a voting method combined with a confidence ranking method to fuse the results.

[0334] <1> Voting method: The recognition result of each target video frame is regarded as a vote, the valid votes for each recognition result are counted, and the recognition result with the most votes is taken as the final output.

[0335] <2> Confidence sorting method: If there are two valid votes, the confidence sorting method is used to determine the final manned result, and the one with the highest confidence is selected as the final result output.

[0336] In the embodiments of the present application, there are at least the following beneficial effects:

[0337] 1) A scoring system based on multiple video frames was constructed: the system precisely focuses on the dynamic frame-to-frame clarity, completeness, and saliency of the target area. This mechanism quantifies the presentation quality of a specific target in consecutive video frames, provides a scientific basis for the selection of the optimal frame, and ensures the high quality of the detection input.

[0338] 2) Accurately locate the optimal frame: A dynamic multi-frame optimization strategy is introduced, which comprehensively considers the video frame quality and inter-frame continuity, and accurately captures the "golden three frames" of the target vehicle from appearance to disappearance, effectively reducing redundant data processing and greatly improving computing efficiency and real-time performance.

[0339] 3) Optimize detection quality: Through joint analysis of multiple frames rather than relying on a single frame, false detections and missed detections caused by adverse factors such as occlusion and blur are significantly reduced, ensuring high accuracy and reliability of detection results.

[0340] 4) Integrated solution: Innovatively integrates three independent tasks of vehicle type recognition, cargo loading status recognition, and cargo occupant recognition into a single recognition model. Through the deep fusion of the shared feature extraction layer and the task-specific layer, it achieves maximum utilization of computing resources and seamless coordination between tasks.

[0341] 5) Soft sharing strategy enhancement: By utilizing the intrinsic connections between tasks and implementing a soft sharing strategy, the independence of feature representation is retained while promoting the effective flow and enhancement of information, achieving a synergistic effect of “1+1+1>3” and significantly improving the generalization ability of the model and its adaptability to complex scenarios.

[0342] 6) Based on the detection results of the best three frames, the logical relationship between the results of the loading of passengers in the cargo bed, the vehicle type results, and the loading status results of the cargo bed are innovatively utilized. At the same time, the voting method and the confidence ranking method are integrated to perform double verification on the results of the loading of passengers in the cargo bed. This not only improves the accuracy of recognition, but also ensures the rationality and stability of the output results, effectively reducing the uncertainty that may be caused by single-frame analysis.

[0343] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for identifying passengers in the cargo box of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0344] The present application also provides a device for identifying people in a truck bed. Fig.13 For the structural diagram of the vehicle loading identification device provided in this application, please refer to Fig.13 The vehicle compartment passenger identification device comprises:

[0345] The detection module 1301 is used to detect N video frames where the target vehicle is located from multiple video frames of the collected video stream; wherein N is an integer greater than 2;

[0346] The scoring module 1302 is used to score the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result; wherein the scoring result represents the reliability of using each video frame to identify the passenger in the vehicle compartment of the target vehicle;

[0347] A screening module 1303 is used to screen at least two target video frames from the N video frames according to the scoring result;

[0348] The recognition module 1304 is used to input the at least two target video frames into a pre-trained recognition model, and the recognition model recognizes the recognition results corresponding to the target video frames;

[0349] The judgment module 1305 is used to judge whether the vehicle compartment of the target vehicle is carrying people based on the recognition results corresponding to the at least two target video frames.

[0350] The vehicle-carrying passenger identification device provided by the present application adopts the vehicle-carrying passenger identification method in the above-mentioned embodiment, and can provide a vehicle-carrying passenger identification method that can be adapted to complex and changeable traffic scenarios. Compared with the prior art, the beneficial effects of the vehicle-carrying passenger identification device provided by the present application are the same as the beneficial effects of the vehicle-carrying passenger identification method provided by the above-mentioned embodiment, and other technical features of the vehicle-carrying passenger identification device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0351] The present application provides a device for identifying passengers in a vehicle bed, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for identifying passengers in the vehicle bed in the above-mentioned embodiment.

[0352] Reference below Fig.14 , Fig.14 A structural diagram of a vehicle-carrying person identification device provided for the present application is shown, which shows a structural diagram of a vehicle-carrying person identification device suitable for implementing an embodiment of the present application. The vehicle-carrying person identification device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The vehicle bed passenger identification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0353] like Fig.14As shown, the vehicle load passenger identification device may include a processing device 1401 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1402 or a program loaded from a storage device 1403 to a random access memory (RAM: Random Access Memory) 1404. Various programs and data required for the operation of the vehicle load passenger identification device are also stored in RAM1404. The processing device 1401, ROM1402, and RAM1404 are connected to each other via a bus 1405. An input / output (I / O) interface 1406 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1406: input devices 1407 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1408 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1403 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1409. The communication device 1409 may allow the vehicle bed passenger identification device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a vehicle bed passenger identification device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0354] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from a storage device 1403, or installed from a ROM 1402. When the computer program is executed by the processing device 1401, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0355] The vehicle-carrying passenger identification device provided by the present application adopts the vehicle-carrying passenger identification method in the above-mentioned embodiment, and can provide a vehicle-carrying passenger identification method that can be adapted to complex and changeable traffic scenarios. Compared with the prior art, the beneficial effects of the vehicle-carrying passenger identification device provided by the present application are the same as the beneficial effects of the vehicle-carrying passenger identification method provided by the above-mentioned embodiment, and the other technical features of the vehicle-carrying passenger identification device are the same as the features disclosed in the method of the previous embodiment, which will not be described in detail here.

[0356] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0357] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0358] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for identifying passengers in a vehicle compartment in the above-mentioned embodiment.

[0359] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0360] The computer-readable storage medium may be included in the identification device for the passenger in the cargo bed; or may exist independently without being assembled into the identification device for the passenger in the cargo bed.

[0361] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the vehicle-carrying passenger identification device, the vehicle-carrying passenger identification device performs the following steps:

[0362] Detecting N video frames where the target vehicle is located from multiple video frames of the collected video stream; wherein N is an integer greater than 2;

[0363] Scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result; wherein the scoring result represents the reliability of using each video frame to identify the passenger in the vehicle compartment of the target vehicle;

[0364] Selecting at least two target video frames from the N video frames according to the scoring results;

[0365] Inputting the at least two target video frames into a pre-trained recognition model respectively, and using the recognition model to identify recognition results corresponding to the target video frames respectively;

[0366] Based on the recognition results respectively corresponding to the at least two target video frames, it is determined whether the vehicle compartment of the target vehicle is carrying people.

[0367] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0368] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0369] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0370] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for identifying passengers in a cargo compartment, and can provide a method for identifying passengers in a cargo compartment that can be adapted to complex and changeable traffic scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the method for identifying passengers in a cargo compartment provided by the above-mentioned embodiment, and will not be elaborated here.

[0371] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for identifying passengers in a vehicle compartment.

[0372] The computer program product provided by the present application can provide a method for identifying passengers in a vehicle bed that can be adapted to complex and changeable traffic scenarios. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the method for identifying passengers in a vehicle bed provided by the above embodiment, and will not be elaborated here.

[0373] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for identifying people in a vehicle bucket, characterized in that: The method comprises: Detecting N video frames where the target vehicle is located from multiple video frames of the collected video stream; wherein N is an integer greater than 2; Scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result; wherein the scoring result represents the reliability of using each video frame to identify the passenger in the vehicle compartment of the target vehicle; Selecting at least two target video frames from the N video frames according to the scoring results; Inputting the at least two target video frames into a pre-trained recognition model respectively, and using the recognition model to identify recognition results corresponding to the target video frames respectively; Based on the recognition results respectively corresponding to the at least two target video frames, it is determined whether the vehicle compartment of the target vehicle is carrying people.

2. The method according to claim 1, characterized in that Scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result includes: Performing an image quality score on each of the N video frames to obtain a target quality comprehensive score corresponding to each video frame; Performing a dynamic information score on each of the N video frames to obtain a dynamic information score corresponding to each video frame; Determine a scoring result corresponding to each video frame based on at least one of a target quality comprehensive score and a dynamic information score corresponding to each video frame; The step of selecting at least two target video frames from the N video frames according to the scoring results includes: At least two target video frames are selected from the N video frames according to the order of the scoring results corresponding to the video frames from high to low.

3. The method according to claim 2, characterized in that The N video frames are a plurality of video frames that are continuous in chronological order, and the step of performing a dynamic information score on each of the N video frames to obtain a dynamic information score corresponding to each video frame includes: The following steps are performed for each of the N video frames: Extracting key dynamic information corresponding to the video frame; wherein the key dynamic information includes at least one of optical flow information and feature point motion information, the optical flow information represents the motion of the feature point in the video frame between two consecutive frames, and the feature point motion information represents the position change of the feature point in the video frame in the time series; Based on the key dynamic information, calculate relevant parameters of the feature point in the video frame; wherein the relevant parameters include at least one of the motion intensity change rate, the complex intensity of the moving path, the speed change amount and the acceleration change amount, the motion intensity change rate represents the change rate between the total displacement of the feature point in the video frame or the modulus of the optical flow vector and the previous video frame, the complex intensity of the moving path represents the ratio of the moving path distance of the feature point in the video frame to the straight-line distance between the starting point and the end point of the moving path, the speed change amount represents the speed change amount of the feature point in the video frame between the m video frames before and after the video frame, and the acceleration change amount represents the acceleration change amount of the feature point in the video frame between the m video frames before and after the video frame, where m is an integer greater than 2; Determining a candidate score of dynamic information corresponding to the video frame based on relevant parameters of the feature points in the video frame; The dynamic information score of the video frame is determined based on the dynamic information candidate scores respectively corresponding to the video frame and the previous video frame.

4. The method according to claim 2, characterized in that The step of performing image quality scoring on each of the N video frames to obtain a target quality comprehensive score corresponding to each video frame comprises: The following steps are performed for each of the N video frames: Evaluating the image clarity of the video frame to obtain a first score; Evaluating the completeness of the target vehicle in the video frame to obtain a second score; evaluating the prominence of the target vehicle in the video frame to obtain a third score; Based on at least two of the first score and the corresponding first weight, the second score and the corresponding second weight, and the third score and the corresponding third weight, a target quality comprehensive score corresponding to the video frame is calculated.

5. The method according to claim 4, characterized in that The step of evaluating the image clarity of the video frame to obtain a first score comprises: Using the Sobel edge detection method, the gradient of the grayscale value of the pixel in the video frame is calculated; wherein the gradient of the grayscale value of the pixel in the video frame is used to characterize the image clarity; Obtaining a first candidate score for the video frame based on a mean value of a gradient of pixel grayscale values ​​in the video frame; Using the Laplacian gradient detection method, the second-order derivative of the grayscale value of the pixel in the video frame is calculated; wherein the second-order derivative of the grayscale value of the pixel in the video frame is used to characterize the image clarity; Obtaining a second candidate score for the video frame by calculating a Laplace gradient variance based on a second-order derivative of a pixel grayscale value in the video frame; A first score is determined based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight.

6. The method according to claim 5, characterized in that The step of determining the first score based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight, comprises: Obtaining a third candidate score for the video frame based on the first candidate score and the corresponding fourth weight, and the second candidate score and the corresponding fifth weight; Performing K-Medoids clustering on the third candidate scores corresponding to the N video frames respectively to obtain K clusters; wherein the difference between the third candidate scores corresponding to the video frames in the same cluster is less than the first value; wherein K represents the number of categories, and K is an integer greater than 1; Sorting the K clusters according to the central value of the third candidate score in each cluster from small to large to obtain ordered K clusters; For each of the K ordered clusters, based on the sequence number of the cluster in the K ordered clusters and K, a clarity score of the cluster is determined as a first score corresponding to all video frames in the cluster.

7. The method according to claim 4, characterized in that The step of evaluating the completeness of the target vehicle in the video frame to obtain a second score comprises: Inputting the video frame into a pre-trained target detection model, and using the target detection model to evaluate the integrity of the target vehicle in the video frame to obtain a second score; The training set used when training the target detection model includes samples with occluded targets and samples with incomplete targets.

8. The method according to claim 4, characterized in that The step of evaluating the prominence of the target vehicle in the video frame to obtain a third score comprises: The video frame is input into a preset saliency recognition model, and the saliency recognition model evaluates the saliency of the target area in the video frame, and outputs the grayscale value corresponding to each pixel in the target area; wherein the target area is the area of ​​the target vehicle in the video frame, and the grayscale value corresponding to each pixel in the target area represents the saliency strength of each pixel in the video frame; A third score is determined based on the sum of the grayscale values ​​corresponding to all pixels in the target area.

9. The method according to any one of claims 1 to 8, characterized in that Before the step of scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result, the method further includes: For each of the N video frames, the following steps are performed: Detect and select the framed area where the target vehicle is located; The area after the framed area is expanded according to a preset scaling ratio is used as the target detection area of ​​the video frame; wherein the scaling ratio is greater than 1; The step of scoring the N video frames according to at least one of the image quality and dynamic information of the N video frames to obtain a scoring result comprises: According to at least one of the image quality and dynamic information of the N video frames, the target detection area of ​​each of the N video frames is scored respectively to obtain a scoring result.

10. The method according to any one of claims 1 to 8, characterized in that: The recognition model includes a vehicle type recognition module, a truck loading recognition module and a truck loading passenger recognition module, and the recognition result includes a first result output by the vehicle type recognition module, a second result output by the truck loading recognition module and a third result output by the truck loading passenger recognition module; The step of inputting the at least two target video frames into a pre-trained recognition model respectively, and using the recognition model to identify the recognition results corresponding to the target video frames respectively, comprises: For each of the at least two target video frames, the following steps are performed: Inputting the target video frame into the vehicle type recognition module, and having the vehicle type recognition module recognize the vehicle type of the target vehicle from the target video frame as the first result; The target video frame is input into the bucket loading recognition module, and the bucket loading recognition module recognizes the bucket loading state of the target vehicle from the target video frame as the second result; wherein the bucket loading state includes the bucket being loaded or the bucket being unloaded; The target video frame is input into the vehicle load recognition module, and the vehicle load recognition module recognizes the vehicle load state of the target vehicle from the target video frame as the third result; wherein the vehicle load state includes the vehicle load being loaded or the vehicle load being unloaded; Among them, the vehicle type recognition module, the truck bed loading recognition module and the truck bed passenger recognition module are trained based on a multi-task learning method.

11. The method according to claim 10, characterized in that Before the step of inputting the at least two target video frames into a pre-trained recognition model respectively, and allowing the recognition model to identify the recognition results corresponding to the target video frames respectively, the method further includes: Using a pre-trained shared feature extractor to extract shared features in a training set, and building a shared backbone network based on the shared features; Using the training set to train a preset first branch network, a second branch network and a third branch network respectively; Based on the shared backbone network and the trained first branch network, a vehicle type recognition candidate network is constructed; based on the shared backbone network and the trained second branch network, a vehicle loading recognition candidate network is constructed; based on the shared backbone network and the trained third branch network, a vehicle loading candidate network is constructed; Using a preset multi-task loss function, the vehicle type recognition candidate network, the truck bed loading recognition candidate network and the truck bed passenger candidate network are trained respectively to obtain the vehicle type recognition module, the truck bed loading recognition module and the truck bed passenger recognition module; wherein the multi-task loss function is the weighted sum of the loss functions of the vehicle type recognition candidate network, the truck bed loading recognition candidate network and the truck bed passenger candidate network.

12. The method according to claim 10, characterized in that The step of judging whether the bucket of the target vehicle is carrying people based on the recognition results respectively corresponding to the at least two target video frames comprises: For each of the at least two target video frames, determining whether a first result corresponding to the target video frame is a target vehicle type with a truck bucket; In the case where it is determined that the first result corresponding to the target video frame is the target vehicle type, it is determined whether the second result corresponding to the target video frame is that the bucket is loaded; In the case where it is determined that the second result corresponding to the target video frame is that the bucket is loaded, the third result corresponding to the target video frame is taken as a valid result; Based on the valid results corresponding to the at least two target video frames, it is determined whether the vehicle compartment of the target vehicle is carrying people.

13. The method according to claim 12, characterized in that The step of judging whether the bucket of the target vehicle is carrying people based on the valid results corresponding to the at least two target video frames comprises: When it is determined that the number of corresponding valid results in the at least two target video frames is an odd number, comparing a first number of valid results indicating that the vehicle bucket is loaded with people and a second number of valid results indicating that the vehicle bucket is not loaded with people, if the first number is greater than the second number, determining that the vehicle bucket of the target vehicle is loaded with people; if the first number is less than the second number, determining that the vehicle bucket of the target vehicle is not loaded with people; When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence among the valid results is greater than or equal to the second value, if the valid result with the highest confidence indicates that the bucket is loaded with people, then it is determined that the bucket of the target vehicle is loaded with people; if the valid result with the highest confidence indicates that the bucket is not loaded with people, then it is determined that the bucket of the target vehicle is not loaded with people; When it is determined that the number of corresponding valid results in the at least two target video frames is an even number and the highest confidence level among the valid results is less than a second value, it is determined that the bucket of the target vehicle is unmanned.

14. A vehicle loading person identification device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for identifying a person in a vehicle compartment as claimed in any one of claims 1 to 13.

15. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for identifying a person in a vehicle compartment as described in any one of claims 1 to 13 are implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method for identifying a person in a vehicle compartment as claimed in any one of claims 1 to 13 are implemented.

Images