Scene recognition method and device, computer device and storage medium

Abstract

The application relates to a scene recognition method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a video stream corresponding to a scene to be recognized; performing trajectory detection on the video stream according to a trained target detection model to obtain a trajectory detection result corresponding to the scene to be recognized; when the trajectory detection result represents that there is a trajectory, randomly extracting a video frame from the video stream and inputting the video frame into a trained scene classification model to obtain a scene classification result corresponding to the video stream; and determining a scene recognition result according to the scene classification result corresponding to the video stream. The method can obtain the scene classification result of the scene to be recognized, realize accurate scene recognition, and thus can determine a compliance scene that needs to be subjected to violent sorting recognition according to the scene classification result, and improve the recognition efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a scene recognition method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of computer technology, violent sorting identification technology has emerged in the logistics field. Violent sorting identification technology is mainly used to accurately and quickly screen out whether there is violent sorting behavior in application scenarios such as warehouses, so as to provide more accurate and timely reminders and guidance.

[0003] In traditional technologies, when performing brute force sorting and recognition, brute force sorting and recognition are mainly performed on scene images captured by cameras used for scene monitoring.

[0004] However, in traditional technologies, since all scene images captured by cameras used for scene monitoring need to be brute-force sorted and recognized, there is a problem of low recognition efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a scene recognition method, device, computer equipment and storage medium that can improve the efficiency of brute force sorting recognition in response to the above technical problems.

[0006] A scene recognition method, the method comprising:

[0007] Obtain the video stream corresponding to the scene to be identified;

[0008] Perform trajectory detection on the video stream based on the trained target detection model to obtain the trajectory detection result corresponding to the scene to be identified;

[0009] When the trajectory detection result indicates that a trajectory exists, a video frame is randomly extracted from the video stream and input into the trained scene classification model to obtain the scene classification result corresponding to the video stream;

[0010] The scene recognition result is determined based on the scene classification result corresponding to the video stream.

[0011] In one embodiment, before performing trajectory detection on the video stream according to the trained object detection model to obtain a trajectory detection result corresponding to the scene to be identified, the method further includes:

[0012] Obtain sample fusion images with violent sorting trajectory annotations;

[0013] The initial target detection model is trained according to the sample fusion image to obtain a trained target detection model.

[0014] In one embodiment, trajectory detection is performed on the video stream based on the trained object detection model, and the trajectory detection results corresponding to the scene to be identified include:

[0015] Perform image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified;

[0016] The fused image is input into the trained target detection model for trajectory detection to obtain the trajectory detection results to be screened;

[0017] Threshold screening is performed on the trajectory detection results to be screened to obtain the trajectory detection results corresponding to the scene to be identified.

[0018] In one embodiment, threshold screening is performed on the trajectory detection results to be screened, and the trajectory detection results corresponding to the scene to be identified are obtained, including:

[0019] The candidate trajectory detection frames in the trajectory detection results are screened according to a preset detection frame threshold to obtain candidate trajectory detection frames corresponding to the scene to be identified;

[0020] According to the confidence of the candidate trajectory detection frame, the candidate trajectory detection frame is selected by IOU (Intersection over Union) threshold to obtain the target trajectory detection frame corresponding to the scene to be identified;

[0021] According to the target trajectory detection frame, the trajectory detection result corresponding to the scene to be identified is obtained.

[0022] In one embodiment, before randomly extracting video frames from a video stream and inputting them into a trained scene classification model to obtain a scene classification result corresponding to the video stream, the method further includes:

[0023] Obtain classified sample images with category labels, including distorted or black screen, containing objects, and not containing objects;

[0024] The initial scene classification model is trained according to the classified sample images to obtain a trained scene classification model.

[0025] In one embodiment, determining a scene recognition result based on a scene classification result corresponding to the video stream includes:

[0026] When the scene classification result corresponding to the video stream is that it contains an object, the preset statistical value is updated, and the step of obtaining the video stream corresponding to the scene to be identified is returned until the preset statistical value is equal to the preset number threshold, and the scene recognition result is determined to be scene compliance.

[0027] In one embodiment, the scene recognition method further includes:

[0028] Periodically extracting the video frames to be identified corresponding to the scene to be identified;

[0029] input the video frame to be identified into the trained scene classification model to obtain a scene classification result corresponding to the video frame to be identified;

[0030] when the scene classification result corresponding to the video frame to be identified is a screen of garbage or a black screen, the scene recognition result is determined as a scene being out of compliance.

[0031] A scene recognition device, the device comprising:

[0032] a obtaining module configured to obtain a video stream corresponding to a scene to be identified;

[0033] a trajectory detection module configured to perform trajectory detection on the video stream according to a trained target detection model to obtain a trajectory detection result corresponding to the scene to be identified;

[0034] a classification module configured to, when the trajectory detection result indicates that there is a trajectory, randomly extract a video frame from the video stream and input the video frame into a trained scene classification model to obtain a scene classification result corresponding to the video stream;

[0035] a processing module configured to determine a scene recognition result according to the scene classification result corresponding to the video stream.

[0036] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0037] obtaining a video stream corresponding to a scene to be identified;

[0038] performing trajectory detection on the video stream according to a trained target detection model to obtain a trajectory detection result corresponding to the scene to be identified;

[0039] when the trajectory detection result indicates that there is a trajectory, randomly extracting a video frame from the video stream and inputting the video frame into a trained scene classification model to obtain a scene classification result corresponding to the video stream;

[0040] determining a scene recognition result according to the scene classification result corresponding to the video stream.

[0041] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0042] obtaining a video stream corresponding to a scene to be identified;

[0043] performing trajectory detection on the video stream according to a trained target detection model to obtain a trajectory detection result corresponding to the scene to be identified;

[0044] When the trajectory detection result indicates that a trajectory exists, a video frame is randomly extracted from the video stream and input into the trained scene classification model to obtain the scene classification result corresponding to the video stream;

[0045] The scene recognition result is determined based on the scene classification result corresponding to the video stream.

[0046] The above-mentioned scene recognition method, device, computer equipment and storage medium use a trained target detection model to perform trajectory detection on the video stream corresponding to the scene to be identified. On the basis of the trajectory detection result being characterized as the existence of a trajectory, the trained scene classification model is further used to perform scene classification on randomly extracted video frames in the video stream. The scene classification result of the scene to be identified can be obtained, and accurate scene recognition can be achieved. Therefore, the compliant scenes that need to be brute-force sorting and identification can be determined based on the scene classification result, thereby improving recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a flow chart of a scene recognition method according to an embodiment;

[0048] Figure 2 is a schematic diagram of a scene recognition method in one embodiment;

[0049] Figure 3 is a schematic flow chart of a scene recognition method in another embodiment;

[0050] Figure 4 is a structural block diagram of a scene recognition device in one embodiment;

[0051] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] In one embodiment, Figure 1 As shown, a scene recognition method is provided. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] Step 102: Obtain a video stream corresponding to the scene to be identified.

[0055] The scene to be identified refers to the scene monitored by the corresponding camera used for scene monitoring. For example, the scene to be identified can be a warehouse. The video stream refers to the video data collected by the camera corresponding to the scene to be identified.

[0056] Specifically, the server obtains the video stream corresponding to the scene to be identified from the camera corresponding to the scene to be identified. Furthermore, the camera corresponding to the scene to be identified collects the video stream in real time and periodically sends the collected video stream to a preset database. The server can directly obtain the video stream from the preset database.

[0057] Step 104 : performing trajectory detection on the video stream according to the trained object detection model to obtain a trajectory detection result corresponding to the scene to be identified.

[0058] Among them, the target detection model refers to the model used to detect the target. In this application, the target refers to the trajectory of the object. The trajectory detection result corresponds to whether there is a trajectory that meets the preset requirements in the video stream. When there is a trajectory that meets the preset requirements in the video stream, the trajectory detection result includes a detection box and the confidence level of the detection box. In this case, the trajectory detection result indicates that the trajectory exists. When there is no trajectory that meets the preset requirements in the video stream, the trajectory detection result is empty. In this case, the trajectory detection result indicates that the trajectory does not exist. The preset requirements can be set as needed.

[0059] Specifically, the server will fuse the video frames in the video stream in chronological order to obtain a fused image, and then use the trained target detection model to perform trajectory detection on the fused image. If there is a trajectory in the fused image, the target detection model will mark multiple detection boxes and the confidence level of each detection box on the fused image. Each detection box will be screened according to its confidence level to obtain the trajectory detection result.

[0060] Specifically, when performing image fusion, the server can use each video frame to obtain the image optical flow trajectory. This application does not limit the method of image fusion. Taking the use of each video frame to obtain the image optical flow trajectory as an example, assuming that 6 consecutive video frames are to be fused, the server will first obtain 6 consecutive video frames, convert the 6 consecutive video frames into grayscale images, and then add the pixels of the first and second images and divide them by 2, add the pixels of the third and fourth images and divide them by 2, and add the pixels of the fifth and sixth images and divide them by 2. The above three results are used as the RGB three channels of the final fusion image.

[0061] Specifically, the target detection network can be a Yolov3 detection network, and the application does not limit the target detection network. Taking the Yolov3 detection network as an example, the Yolov3 detection network first extracts features by taking darknet-53 as a feature extraction network, then uses a continuous 3x3 and 1x1 convolution layer and a number of shortcut connections, mainly predicts four bounding box coordinates, bounding box confidence and the possibility of each class (in this application, it mainly involves the trajectory class), and the Yolov3 uses three feature maps for prediction. The first feature map is 13x13, the second is 26x26, and the third is 52x52. Each feature map pixel predicts 3 detection boxes. The receptive field of the three output layers of the Yolov3 is about (85x85, 181x181, 365x365). The three feature maps can detect objects of different sizes. In the case where the object sizes are similar, only one layer of prediction is needed, which can greatly reduce the network parameter amount.

[0062] Step 106, when the trajectory detection result represents that there is a trajectory, randomly extracting a video frame from the video stream inputs the trained scene classification model to obtain a scene classification result corresponding to the video stream.

[0063] Specifically, when the trajectory detection result represents that there is a trajectory, the server will randomly extract a video frame from the video stream, and according to the detection box in the trajectory detection result, the video frame is cut to obtain a to-be-input image. The to-be-input image is input into the trained scene classification model to obtain a scene classification result corresponding to the video stream. The trained scene classification model can be used for scene recognition of the to-be-input image. Possible scene categories include objects and do not include objects. In addition to classifying whether to include objects, the scene classification model can also be used to classify whether the video frame is a screen or a black screen. Wherein, the object includes express delivery, package, etc.

[0064] Specifically, the application does not limit the scene classification model, and preferably, the scene classification model can be an EfficientNet-d0 classification model. Taking the EfficientNet-d0 classification model as an example, when the trajectory detection result represents that there is a trajectory, the server will perform picture cutting on the randomly extracted video frame according to the detection box in the trajectory detection result to obtain a to-be-input image. Then the to-be-input image is input into the trained EfficientNet-d0 classification model to obtain a scene classification result corresponding to the video stream. Compared with other existing convolutional neural networks on ImageNet, the EfficientNet-d0 classification model has higher accuracy and higher efficiency for the same size input, and can reduce the parameter size and FLOPS order of magnitude.

[0065] Step 108: Determine a scene recognition result based on the scene classification result corresponding to the video stream.

[0066] Scene recognition results include "compliant" and "non-compliant" scenarios. A "compliant" scenario means the scene to be identified meets the requirements and involves throwing or sorting. Violent sorting and recognition are required on the scene images captured by the corresponding camera. A "non-compliant" scenario means the scene to be identified does not meet the requirements and does not involve throwing or sorting. Violent sorting and recognition are not required on the scene images captured by the corresponding camera.

[0067] Specifically, when the scene classification result corresponding to the video stream is that it does not contain an object, the server can directly determine that the scene recognition result is a non-compliant scene. When the scene classification result corresponding to the video stream is that it contains an object, the server needs to count the scene classification results, and then return to the step of obtaining the video stream corresponding to the scene to be identified, and further judge. When the statistical value of the statistical scene classification result containing an object reaches a preset number threshold, the scene recognition result is determined to be a compliant scene. Otherwise, the scene recognition result is considered to be non-compliant.

[0068] The above-mentioned scene recognition method uses a trained target detection model to perform trajectory detection on the video stream corresponding to the scene to be identified. On the basis of the trajectory detection result being characterized as the existence of a trajectory, the trained scene classification model is further used to perform scene classification on randomly extracted video frames in the video stream. The scene classification result of the scene to be identified can be obtained, and accurate scene recognition can be achieved. Therefore, the compliant scenes that need to be brute-force sorting and identification can be determined based on the scene classification result, thereby improving recognition efficiency.

[0069] In one embodiment, before performing trajectory detection on the video stream according to the trained object detection model to obtain a trajectory detection result corresponding to the scene to be identified, the method further includes:

[0070] Obtain sample fusion images with violent sorting trajectory annotations;

[0071] The initial target detection model is trained according to the sample fusion image to obtain a trained target detection model.

[0072] Brute force sorting trajectory annotation refers to marking the trajectory of the brute force sorted objects on the sample fused image. The sample fused image is the image obtained by fusing consecutive video frames from the sample video stream. The sample video stream refers to video data that has been determined to have undergone brute force sorting. The sample video stream and sample fused image can be pre-stored in a preset database.

[0073] Specifically, the server can obtain a sample fusion image with a brute force sorting trajectory annotation from a preset database, use the sample fusion image as input, and use the brute force sorting trajectory annotation carried by the sample fusion image as a label to perform supervised training on the initial target detection model to obtain a trained target detection model. Taking the target detection model as an example of the Yolov3 detection network, the server can preferably change the image size to 608X416 and use the 32 sample fusion images as a training set for backpropagation iterative training. The Yolov3 detection network will output multiple box predictions, and each box needs to calculate the bounding box coordinate loss, bounding box confidence loss, and category classification loss.

[0074] In this embodiment, by obtaining a sample fusion image carrying violent sorting trajectory annotations, the initial target detection model is trained according to the sample fusion image to obtain a trained target detection model, thereby achieving training of the target detection model.

[0075] In one embodiment, trajectory detection is performed on the video stream based on the trained object detection model, and the trajectory detection results corresponding to the scene to be identified include:

[0076] Perform image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified;

[0077] The fused image is input into the trained target detection model for trajectory detection to obtain the trajectory detection results to be screened;

[0078] Threshold screening is performed on the trajectory detection results to be screened to obtain the trajectory detection results corresponding to the scene to be identified.

[0079] The track detection results to be screened refer to the detection results output by the target detection model after performing track detection based on the fused image. The track detection results to be screened include a large number of detection boxes and the confidence levels of the detection boxes.

[0080] Specifically, when performing trajectory detection, the server will perform image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified, and then input the fused image into the trained target detection model for trajectory detection. The target detection model will output multiple detection frames and the confidence of the detection frames based on the fused image. The server will use the multiple detection frames and the confidence of the detection frames output by the target detection model as the trajectory detection results to be screened, and perform threshold screening on the trajectory detection results to be screened to obtain the trajectory detection results corresponding to the scene to be identified. Among them, screening mainly refers to screening using the preset detection frame threshold, IOU threshold and detection frame confidence. Through this screening method, detection frames with confidence greater than the preset detection frame threshold are screened out to achieve accurate trajectory detection. Among them, the preset detection frame threshold can be set as needed.

[0081] In this embodiment, by performing image fusion on each video frame in the video stream, the fused image is input into the trained target detection model for trajectory detection to obtain the trajectory detection results to be screened, and the trajectory detection results to be screened are threshold-screened to obtain the trajectory detection results corresponding to the scene to be identified, so that accurate trajectory detection results can be obtained.

[0082] In one embodiment, threshold screening is performed on the trajectory detection results to be screened, and the trajectory detection results corresponding to the scene to be identified are obtained, including:

[0083] The candidate trajectory detection frames in the trajectory detection results are screened according to a preset detection frame threshold to obtain candidate trajectory detection frames corresponding to the scene to be identified;

[0084] According to the confidence of the candidate trajectory detection frame, the candidate trajectory detection frame is selected by IOU threshold to obtain the target trajectory detection frame corresponding to the scene to be identified;

[0085] According to the target trajectory detection frame, the trajectory detection result corresponding to the scene to be identified is obtained.

[0086] Specifically, such as Figure 2 As shown in the figure, the server will first detect the frame based on the preset threshold (i.e. Figure 2 The score value threshold in the filter is used as well as the confidence of the track detection frame to be filtered in the track detection result to be filtered. The track detection frame to be filtered is filtered to obtain the candidate track detection frame corresponding to the scene to be identified (i.e. Figure 2 The remaining part of the frame in the candidate trajectory detection frame is selected by IOU threshold according to the confidence of the candidate trajectory detection frame to determine the target trajectory detection frame (i.e. Figure 2 The target trajectory detection frame is taken as the trajectory detection result corresponding to the scene to be identified. Preferably, the score value threshold can be 0.2.

[0087] Specifically, the IOU threshold selection method can be: sort the candidate trajectory detection frames according to their confidence, select the candidate trajectory detection frame with the highest score, and traverse the remaining candidate trajectory detection frames based on the candidate trajectory detection frame with the highest score. If there is a detection frame in the remaining candidate trajectory detection frames whose detection frame overlaps with the candidate trajectory detection frame with the highest score by more than a preset ratio threshold, it will be deleted, and the candidate trajectory detection frame with the highest score will be selected from the remaining candidate trajectory detection frames that have not been deleted. The step of traversing the remaining candidate trajectory detection frames based on the highest score is returned until all candidate trajectory detection frames are selected and the target trajectory detection frame corresponding to the scene to be identified is obtained. The preset ratio threshold can be set as needed. In this way, redundant detection frames in the candidate trajectory detection frames can be selected.

[0088] In this embodiment, by screening the trajectory detection frames to be screened according to the preset detection frame threshold and the IOU threshold, it is possible to achieve the selection of redundant detection frames while reducing the computational complexity of the IOU threshold selection.

[0089] In one embodiment, before randomly extracting video frames from a video stream and inputting them into a trained scene classification model to obtain a scene classification result corresponding to the video stream, the method further includes:

[0090] Obtain classified sample images with category labels, including distorted or black screen, containing objects, and not containing objects;

[0091] The initial scene classification model is trained according to the classified sample images to obtain a trained scene classification model.

[0092] Specifically, the server retrieves sample images with category labels from a pre-set database. Using these sample images as input and the category labels as labels, the server performs supervised training on the initial scene classification model, ultimately generating a trained scene classification model through backpropagation. Furthermore, during training, all sample images can be resized to a uniform size, and the training set can be expanded through common data augmentation methods such as rotation, translation, and noise addition.

[0093] The category label of the classification sample image containing objects can be specifically an image obtained by image cutting on the sample fusion image according to the violent sorting trajectory label. The image cutting can be performed as follows: according to the violent sorting trajectory label, an image region corresponding to the violent sorting trajectory label is cut out from an original image before the sample fusion image is fused, and the image region contains objects such as goods thrown by people. The category label of the classification sample image not containing objects can be specifically an image obtained by image cutting on the sample fusion image, but the image region not corresponding to the violent sorting trajectory label is cut out this time.

[0094] In this embodiment, by obtaining the classification sample image carrying the category label, the category label includes a screen of flowers or a black screen, contains objects, and does not contain objects, the initial scene classification model is trained according to the classification sample image, and a trained scene classification model is obtained, so that the training of the scene classification model can be realized.

[0095] In one embodiment, according to the scene classification result corresponding to the video stream, the scene recognition result is determined, including:

[0096] When the scene classification result corresponding to the video stream contains objects, the preset statistical value is updated, and the step of obtaining the video stream corresponding to the scene to be recognized is returned until the preset statistical value is equal to the preset number threshold, and the scene recognition result is determined to be a scene compliance.

[0097] Specifically, when the scene classification result corresponding to the video stream contains objects, it indicates that the scene to be recognized can be a compliant scene, the server updates the preset statistical value, and returns to the step of obtaining the video stream corresponding to the scene to be recognized, and continues to perform scene recognition on the scene to be recognized until the preset statistical value is equal to the preset number threshold, and the server determines that the scene recognition result is a scene compliance. If the preset statistical value is less than the preset number threshold within the preset statistical time, the server can determine that the scene recognition result is a scene non-compliance. The preset statistical value can be set as needed, for example, the initial value of the preset statistical value can be 0, and the preset statistical time can also be set as needed, for example, the preset statistical time can be one day.

[0098] In this embodiment, by updating the preset statistical value and returning to the step of obtaining the video stream corresponding to the scene to be recognized until the preset statistical value is equal to the preset number threshold, the scene recognition result is determined to be a scene compliance, so that accurate scene recognition can be realized.

[0099] In one embodiment, the scene recognition method further includes:

[0100] Periodically extracting the video frame to be recognized corresponding to the scene to be recognized;

[0101] Input the video frame to be identified into the trained scene classification model to obtain the scene classification result corresponding to the video frame to be identified;

[0102] When the scene classification result corresponding to the video frame to be identified is a distorted screen or a black screen, the scene recognition result is obtained as a non-compliant scene.

[0103] Specifically, during scene recognition, the server periodically extracts video frames corresponding to the scene to be recognized, feeds these frames into a trained scene classification model, and obtains the scene classification results corresponding to the frames. If the scene classification result for the frame is a distorted or black screen, the scene recognition result is directly concluded as non-compliant. The extraction period can be customized as needed.

[0104] In this embodiment, the video frames to be identified corresponding to the scene to be identified are periodically extracted, and the video frames to be identified are input into the trained scene classification model to obtain the scene classification results corresponding to the video frames to be identified. When the scene classification results corresponding to the video frames to be identified are a flowery screen or a black screen, the scene recognition result is that the scene is not compliant, and the video frames to be identified can be used to realize scene recognition.

[0105] In one embodiment, Figure 3 As shown, a flow chart is used to illustrate the scene recognition method of the present application.

[0106] The server obtains the video stream corresponding to the scene to be identified, obtains the sample fusion image with the violent sorting trajectory annotation, trains the initial target detection model based on the sample fusion image, obtains the trained target detection model, performs image fusion on each video frame in the video stream, obtains the fusion image corresponding to the scene to be identified, inputs the fusion image into the trained target detection model for trajectory detection, and obtains the trajectory detection result to be screened (corresponding to Figure 3 The trajectory detection module in the trajectory detection module) is used to filter the trajectory detection frame to be filtered in the trajectory detection result according to the preset detection frame threshold, and the candidate trajectory detection frame corresponding to the scene to be identified is obtained. The candidate trajectory detection frame is brushed with the IOU threshold according to the confidence of the candidate trajectory detection frame to obtain the target trajectory detection frame corresponding to the scene to be identified. According to the target trajectory detection frame, the trajectory detection result corresponding to the scene to be identified is obtained (corresponding to Figure 3 The post-processing module in the image processing module is used to obtain classification sample images with category labels, including flower screen or black screen, containing objects and not containing objects. The initial scene classification model is trained based on the classification sample images to obtain the trained scene classification model. When the trajectory detection result indicates that there is a trajectory (i.e. Figure 3 The generated trajectory branch in the video stream is randomly extracted from the video stream and input into the trained scene classification model to obtain the scene classification result corresponding to the video stream (corresponding to Figure 3 The classification module in the video stream) is used to classify the scene containing the object (corresponding to Figure 3 When the preset statistical value is equal to the preset number threshold (corresponding to the number of times the preset statistical value is equal to the preset ... Figure 3 At the same time, the server will periodically extract the video frames to be identified corresponding to the scene to be identified, input the video frames to be identified into the trained scene classification model, and obtain the scene classification results corresponding to the video frames to be identified (corresponding to Figure 3 In the classification module), when the scene classification result corresponding to the video frame to be identified is a flower screen or a black screen, the scene recognition result is obtained as a non-compliant scene.

[0107] It should be understood that although Figure 1 and Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 and Figure 3 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0108] In one embodiment, Figure 4 As shown, a scene recognition device is provided, including: an acquisition module 402, a trajectory detection module 404, a classification module 406 and a processing module 408, wherein:

[0109] An acquisition module 402 is configured to acquire a video stream corresponding to a scene to be identified;

[0110] The trajectory detection module 404 is used to perform trajectory detection on the video stream according to the trained target detection model to obtain a trajectory detection result corresponding to the scene to be identified;

[0111] The classification module 406 is configured to randomly extract video frames from the video stream and input them into the trained scene classification model to obtain a scene classification result corresponding to the video stream when the trajectory detection result indicates the presence of a trajectory;

[0112] The processing module 408 is configured to determine a scene recognition result based on the scene classification result corresponding to the video stream.

[0113] The above-mentioned scene recognition device uses a trained target detection model to perform trajectory detection on the video stream corresponding to the scene to be identified. On the basis of the trajectory detection result indicating the existence of a trajectory, it further uses a trained scene classification model to perform scene classification on randomly extracted video frames in the video stream. The scene classification result of the scene to be identified can be obtained, and accurate scene recognition can be achieved. Therefore, the compliant scenes that need to be brute-force sorting and identification can be determined based on the scene classification result, thereby improving recognition efficiency.

[0114] In one embodiment, the scene recognition device further includes a first model training module, which is used to obtain a sample fusion image carrying violent sorting trajectory annotations, train the initial target detection model based on the sample fusion image, and obtain a trained target detection model.

[0115] In one embodiment, the trajectory detection module is further used to perform image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified, input the fused image into the trained target detection model to perform trajectory detection to obtain a trajectory detection result to be screened, and perform threshold screening on the trajectory detection result to be screened to obtain a trajectory detection result corresponding to the scene to be identified.

[0116] In one embodiment, the trajectory detection module is further used to filter the trajectory detection frames to be filtered in the trajectory detection results according to a preset detection frame threshold to obtain an alternative trajectory detection frame corresponding to the scene to be identified, and to perform IOU threshold selection on the alternative trajectory detection frame according to the confidence of the alternative trajectory detection frame to obtain a target trajectory detection frame corresponding to the scene to be identified, and to obtain a trajectory detection result corresponding to the scene to be identified based on the target trajectory detection frame.

[0117] In one embodiment, the scene recognition device also includes a second model training module, which is used to obtain classification sample images carrying category labels, the category labels including flowery screen or black screen, containing objects and not containing objects, and train the initial scene classification model according to the classification sample images to obtain a trained scene classification model.

[0118] In one embodiment, the processing module is also used to update the preset statistical value when the scene classification result corresponding to the video stream is that it contains an object, and return to the step of obtaining the video stream corresponding to the scene to be identified until the preset statistical value is equal to the preset number threshold, and determine that the scene recognition result is scene compliance.

[0119] In one embodiment, the scene recognition device also includes a recognition module, which is used to periodically extract video frames to be recognized corresponding to the scene to be recognized, input the video frames to be recognized into a trained scene classification model, and obtain scene classification results corresponding to the video frames to be recognized. When the scene classification result corresponding to the video frames to be recognized is a flowery screen or a black screen, the scene recognition result is that the scene is non-compliant.

[0120] For the specific definition of the scene recognition device, please refer to the definition of the scene recognition method above, which will not be repeated here. The various modules in the above-mentioned scene recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0121] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video streams, sample fusion images and classified sample images corresponding to the scene to be identified. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a scene recognition method is implemented.

[0122] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0124] Obtain the video stream corresponding to the scene to be identified;

[0125] Perform trajectory detection on the video stream based on the trained target detection model to obtain the trajectory detection result corresponding to the scene to be identified;

[0126] When the trajectory detection result represents that there is a trajectory, a video frame is randomly extracted from the video stream and input into the trained scene classification model to obtain a scene classification result corresponding to the video stream;

[0127] According to the scene classification result corresponding to the video stream, a scene recognition result is determined.

[0128] The scene recognition computer device, by using the trained target detection model to perform trajectory detection on the video stream corresponding to the scene to be recognized, and on the basis that the trajectory detection result represents that there is a trajectory, further using the trained scene classification model to perform scene classification on the randomly extracted video frame in the video stream, can obtain the scene classification result of the scene to be recognized, realize accurate scene recognition, so that the compliance scene that needs to be identified by violent sorting can be determined according to the scene classification result, and the identification efficiency is improved.

[0129] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining a sample fusion image carrying a violent sorting trajectory label; training the initial target detection model according to the sample fusion image to obtain the trained target detection model.

[0130] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing image fusion on each video frame in the video stream to obtain a fusion image corresponding to the scene to be recognized; inputting the fusion image into the trained target detection model to perform trajectory detection to obtain a to-be-screened trajectory detection result; performing threshold screening on the to-be-screened trajectory detection result to obtain a trajectory detection result corresponding to the scene to be recognized.

[0131] In one embodiment, when the processor executes the computer program, the following steps are also implemented: screening the to-be-screened trajectory detection frame in the to-be-screened trajectory detection result according to a preset detection frame threshold to obtain a candidate trajectory detection frame corresponding to the scene to be recognized; performing IOU threshold pruning on the candidate trajectory detection frame according to the confidence of the candidate trajectory detection frame to obtain a target trajectory detection frame corresponding to the scene to be recognized; and obtaining the trajectory detection result corresponding to the scene to be recognized according to the target trajectory detection frame.

[0132] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining a classification sample image carrying a category label, the category label including a flower screen or a black screen, containing an object, and not containing an object; training the initial scene classification model according to the classification sample image to obtain the trained scene classification model.

[0133] In one embodiment, the processor, when executing the computer program, also implements the following steps: when the scene classification result corresponding to the video stream contains an object, updating a preset statistical value, and returning to the step of obtaining the video stream corresponding to the scene to be identified until the preset statistical value is equal to a preset number threshold, determining that the scene recognition result is a compliant scene.

[0134] In one embodiment, the processor, when executing the computer program, also implements the following steps: periodically extracting a video frame to be identified corresponding to the scene to be identified; inputting the video frame to be identified into the trained scene classification model to obtain a scene classification result corresponding to the video frame to be identified; and when the scene classification result corresponding to the video frame to be identified is a screen of garbage or a black screen, obtaining a scene recognition result that the scene is non-compliant.

[0135] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program, when executed by a processor, implements the following steps:

[0136] obtaining a video stream corresponding to a scene to be identified;

[0137] performing trajectory detection on the video stream according to a trained target detection model to obtain a trajectory detection result corresponding to the scene to be identified;

[0138] when the trajectory detection result indicates that there is a trajectory, inputting a video frame randomly extracted from the video stream into the trained scene classification model to obtain a scene classification result corresponding to the video stream;

[0139] determining a scene recognition result according to the scene classification result corresponding to the video stream.

[0140] The above scene recognition storage medium, by performing trajectory detection on the video stream corresponding to the scene to be identified by using the trained target detection model, and further performing scene classification on the video frame randomly extracted from the video stream by using the trained scene classification model on the basis that the trajectory detection result indicates that there is a trajectory, can obtain the scene classification result of the scene to be identified, realize accurate scene recognition, and thus can determine a compliant scene that needs to be subjected to violent sorting identification according to the scene classification result, and improve the identification efficiency.

[0141] In one embodiment, the computer program, when executed by the processor, also implements the following steps: obtaining a sample fusion image carrying a violent sorting trajectory label; and training an initial target detection model according to the sample fusion image to obtain the trained target detection model.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: performing image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified; inputting the fused image into a trained target detection model to perform trajectory detection to obtain a trajectory detection result to be screened; and performing threshold screening on the trajectory detection result to be screened to obtain a trajectory detection result corresponding to the scene to be identified.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the trajectory detection frames to be filtered in the trajectory detection results to be filtered are filtered according to a preset detection frame threshold to obtain candidate trajectory detection frames corresponding to the scene to be identified; the candidate trajectory detection frames are screened according to the IOU threshold based on the confidence of the candidate trajectory detection frames to obtain the target trajectory detection frame corresponding to the scene to be identified; and the trajectory detection result corresponding to the scene to be identified is obtained according to the target trajectory detection frame.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining classification sample images carrying category labels, the category labels including flowery screen or black screen, containing objects, and not containing objects; training the initial scene classification model based on the classification sample images to obtain a trained scene classification model.

[0145] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the scene classification result corresponding to the video stream is that it contains an object, the preset statistical value is updated, and the step of obtaining the video stream corresponding to the scene to be identified is returned until the preset statistical value is equal to the preset number threshold, and the scene recognition result is determined to be scene compliance.

[0146] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: periodically extracting video frames to be identified corresponding to the scene to be identified; inputting the video frames to be identified into a trained scene classification model to obtain scene classification results corresponding to the video frames to be identified; when the scene classification result corresponding to the video frames to be identified is a distorted screen or a black screen, the scene recognition result is obtained as scene non-compliance.

[0147] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0148] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A scene recognition method, characterized in that: The method comprises: Obtain the video stream corresponding to the scene to be identified; Performing image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified; Inputting the fused image into a trained target detection model to perform trajectory detection to obtain a trajectory detection result to be screened; Performing threshold screening on the trajectory detection results to be screened to obtain trajectory detection results corresponding to the scene to be identified; When the trajectory detection result indicates that a trajectory exists, randomly extracting a video frame from the video stream and inputting it into a trained scene classification model to obtain a scene classification result corresponding to the video stream; A scene recognition result is determined according to the scene classification result corresponding to the video stream.

2. The method according to claim 1, characterized in that Before performing trajectory detection on the video stream according to the trained target detection model to obtain a trajectory detection result corresponding to the scene to be identified, the method further includes: Obtain sample fusion images with violent sorting trajectory annotations; The initial target detection model is trained according to the sample fusion image to obtain a trained target detection model.

3. The method according to claim 1, characterized in that The threshold screening of the to-be-screened trajectory detection results to obtain a trajectory detection result corresponding to the to-be-recognized scene includes: The candidate trajectory detection frames in the trajectory detection results are filtered according to a preset detection frame threshold to obtain candidate trajectory detection frames corresponding to the scene to be identified; Perform IOU threshold selection on the candidate trajectory detection frame according to the confidence of the candidate trajectory detection frame to obtain the target trajectory detection frame corresponding to the scene to be identified; According to the target trajectory detection frame, a trajectory detection result corresponding to the scene to be identified is obtained.

4. The method according to claim 1, wherein Before randomly extracting video frames from the video stream and inputting them into the trained scene classification model to obtain a scene classification result corresponding to the video stream, the method further includes: Obtaining classified sample images with category labels, wherein the category labels include a distorted or black screen, an object, or no object; The initial scene classification model is trained according to the classification sample images to obtain a trained scene classification model.

5. The method according to claim 1, wherein Determining a scene recognition result according to the scene classification result corresponding to the video stream includes: When the scene classification result corresponding to the video stream is that it contains an object, the preset statistical value is updated, and the step of obtaining the video stream corresponding to the scene to be identified is returned until the preset statistical value is equal to the preset number threshold, and the scene recognition result is determined to be scene compliance.

6. The method according to claim 1, characterized in that Also includes: Periodically extracting the video frames to be identified corresponding to the scene to be identified; Inputting the video frame to be identified into a trained scene classification model to obtain a scene classification result corresponding to the video frame to be identified; When the scene classification result corresponding to the video frame to be identified is a distorted screen or a black screen, the scene recognition result is that the scene is not compliant.

7. A scene recognition device, characterized in that: The device comprises: An acquisition module, used to acquire a video stream corresponding to a scene to be identified; a trajectory detection module, configured to perform image fusion on each video frame in the video stream to obtain a fused image corresponding to the scene to be identified, input the fused image into a trained target detection model to perform trajectory detection to obtain a trajectory detection result to be screened, and perform threshold screening on the trajectory detection result to obtain a trajectory detection result corresponding to the scene to be identified; a classification module, configured to, when the trajectory detection result indicates the presence of a trajectory, randomly extract video frames from the video stream and input them into a trained scene classification model to obtain a scene classification result corresponding to the video stream; The processing module is used to determine a scene recognition result according to the scene classification result corresponding to the video stream.

8. The device according to claim 7, characterized in that It also includes a first model training module, which is used to obtain a sample fusion image carrying violent sorting trajectory annotations, train the initial target detection model according to the sample fusion image, and obtain a trained target detection model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images