Video image processing method and target detection model training method
By collaboratively processing video images through edge devices and servers, and automatically identifying and updating deep learning models, the problems of low efficiency and unclear iteration effects in intelligent video analysis systems are solved, and efficient video image analysis and model optimization are achieved.
Patent Information
- Application Number
- CN202510782495.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-12
- Publication Date
- 2025-09-23
Smart Images
Figure CN120689596A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video image processing, and in particular to a video image processing method and a training method for a target detection model. Background Art
[0002] Under various intranet surveillance cameras, people can use intelligent video analysis technology to analyze various security incidents, thereby conducting some production safety management and control; in related technologies, due to network bandwidth limitations, intelligent video analysis systems are usually deployed on edge devices to analyze the video streams output by the edge devices. During the edge analysis process, deep learning models are usually used for image recognition. Therefore, deep learning models are the determining factor in determining the accuracy of the final video analysis results. However, in related technologies, the intelligent video analysis system and the deep learning model training system are two independent systems. From model training, model online to model application, manual offline operations are basically required, which reduces the efficiency of video image analysis. Moreover, when abnormal targets need to be selected as new training samples to iteratively update the deep learning model, manual screenshots of the abnormal targets on the video images are required. Due to the high repetition of continuous sequence images, it is difficult to ensure the quality of the training samples obtained by users through screenshots, resulting in unclear iterative effects on the deep learning model. Summary of the Invention
[0003] The purpose of the present invention is to provide a video image processing method and a target detection model training method to improve the efficiency of video image analysis and improve the iterative effect of deep learning models.
[0004] A video image processing method provided by the present invention is applied to an edge device, and the method includes: obtaining multiple frames of video images based on an original video stream; wherein each frame of the video image includes at least one target object; inputting each frame of the video image into a pre-trained target detection model, and outputting target detection results corresponding to each target object in each frame of the video image; wherein the target detection results include: a target box and detection information corresponding to the target object; the detection information is used to indicate whether the target object has abnormal behavior; based on multiple target detection results, determining at least one abnormal image sequence; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; for each abnormal image sequence, according to a preset optimal image determination method, screening out a target abnormal image from the abnormal image sequence; if an error judgment indication for the target abnormal image is received, sending the target abnormal image to a server, so that the server updates the weight parameters of the target detection model.
[0005] Furthermore, based on multiple target detection results, the step of determining at least one abnormal image sequence includes: taking the first frame of video image as the current video image, and if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior, performing target tracking processing on the abnormal target corresponding to the abnormal detection information; determining the abnormal image sequence based on the image area corresponding to the target box of the abnormal target in the current video image and other video images after the current video image; taking the next frame of video image as the new current video image, and repeating the step of performing abnormal detection information if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior, until the current video image is the last frame of video image.
[0006] Furthermore, for each abnormal image sequence, according to a preset optimal image determination method, the step of screening out a target abnormal image from the abnormal image sequence includes: for each abnormal image sequence, for each abnormal image in the abnormal image sequence, obtaining multiple image evaluation index data corresponding to the abnormal image, performing weighted calculation on the multiple image evaluation index data according to a preset weight coefficient to obtain a weighted result corresponding to the abnormal image; the multiple image evaluation index data include: gradient richness data, image average brightness value, and number of pixels; the abnormal image corresponding to the weighted result with the largest value among the multiple weighted results is determined as the target abnormal image; and the target abnormal image is screened out from the abnormal image sequence.
[0007] Furthermore, if an error judgment indication is received for the target abnormal image, the target abnormal image is sent to the server to update the weight parameters of the target detection model through the server, including: generating an alarm message for the target abnormal image; if an error judgment indication is received for the alarm message, the target abnormal image is sent to the server to update the reflow sample set in the server, and updating the weight parameters of the target detection model based on the reflow sample set through the server.
[0008] Furthermore, the gradient richness data is determined by: calculating the gradient score of each pixel in the abnormal image according to a preset gradient method; and summing the gradient scores of each pixel to obtain the gradient richness data.
[0009] A training method for a target detection model provided by the present invention is applied to a server, and the method includes: determining a sample image based on a preset sample set; wherein the sample set includes: a sample subset collected by an edge device; the sample image includes at least one sample target, and each sample target is marked with a standard frame and a standard detection result; inputting the sample image into an initial model to output a predicted frame and a predicted detection result corresponding to the sample image through the initial model; determining a loss value based on the standard frame, the predicted frame, the standard detection result and the predicted detection result, and updating the initial model based on the loss value until the loss value converges to obtain a target detection model.
[0010] Furthermore, the step of updating the initial model based on the loss value until the loss value converges to obtain the target detection model includes: updating the initial model based on the loss value until the loss value converges to obtain the first detection model; converting the first detection model to obtain the target detection model.
[0011] The present invention provides a video image processing system, which includes: an edge device and a server; the server is used to adopt any of the above-mentioned target detection model training methods to obtain a target detection model, and send the target detection model to the edge device; the edge device is used to adopt any of the above-mentioned video image processing methods based on the target detection model to screen out target abnormal images, and if an error judgment indication for the target abnormal image is received, the target abnormal image is sent to the server; the server is used to update a reflow sample set according to the target abnormal image, and update the weight parameters of the target detection model based on the reflow sample set.
[0012] A video image processing device provided by the present invention is arranged on an edge device, and the device includes: an acquisition module, which is used to acquire multiple frames of video images based on the original video stream; wherein each frame of the video image includes at least one target object; a first output module, which is used to input each frame of the video image into a pre-trained target detection model, and output the target detection results corresponding to each target object in each frame of the video image; wherein the target detection results include: a target box and detection information corresponding to the target object; the detection information is used to indicate whether the target object has abnormal behavior; a first determination module, which is used to determine at least one abnormal image sequence based on multiple target detection results; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; a screening module, which is used to screen out target abnormal images from the abnormal image sequence according to a preset optimal image determination method for each abnormal image sequence; and an update module, which is used to send the target abnormal image to a server if an error judgment indication for the target abnormal image is received, so as to update the weight parameters of the target detection model through the server.
[0013] The present invention provides a training device for a target detection model, which is arranged on a server, and the device includes: a second determination module, which is used to determine a sample image based on a preset sample set; wherein the sample set includes: a sample subset collected by an edge device; the sample image includes at least one sample target, and each sample target is marked with a standard box and a standard detection result; a second output module, which is used to input the sample image into an initial model, so as to output a prediction box and a prediction detection result corresponding to the sample image through the initial model; a third determination module, which is used to determine a loss value based on the standard box, the prediction box, the standard detection result and the prediction detection result, so as to update the initial model based on the loss value until the loss value converges to obtain a target detection model.
[0014] The video image processing method and the target detection model training method provided by the present invention obtain multiple frames of video images based on the original video stream; wherein each frame of the video image includes at least one target object; each frame of the video image is input into a pre-trained target detection model, and the target detection results corresponding to each target object in each frame of the video image are output; wherein the target detection results include: a target box and detection information corresponding to the target object; the detection information is used to indicate whether the target object has abnormal behavior; based on multiple target detection results, at least one abnormal image sequence is determined; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; for each abnormal image sequence, according to a preset optimal image determination method, a target abnormal image is screened out from the abnormal image sequence; if an error judgment indication for the target abnormal image is received, the target abnormal image is sent to a server to update the weight parameters of the target detection model through the server. In this method, the target abnormal image is screened according to a preset optimal image determination method. After receiving an error judgment indication for the target abnormal image, the weight parameters of the target detection model can be updated based on the target abnormal image to continue detecting the video image using the updated target detection model, thereby improving the iterative effect of the target detection model and improving the efficiency of video image analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0016] Figure 1 A flowchart of a video image processing method provided by an embodiment of the present invention;
[0017] Figure 2 A schematic diagram of an abnormal image sequence provided by an embodiment of the present invention;
[0018] Figure 3 A flowchart of a method for training a target detection model provided by an embodiment of the present invention;
[0019] Figure 4 A schematic diagram of a video image processing system provided by an embodiment of the present invention;
[0020] Figure 5 A schematic structural diagram of a video image processing device provided by an embodiment of the present invention;
[0021] Figure 6 A schematic diagram of the structure of a training device for a target detection model provided by an embodiment of the present invention;
[0022] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0024] Intelligent video analysis is to decode the video stream into images and then use artificial intelligence algorithms to identify and classify the images. Under various intranet surveillance cameras, people can use intelligent video analysis technology to analyze various security events, thereby conducting some production safety management and control. In the intelligent video analysis process, in addition to using streaming media technology to receive and decode video streams, it also uses deep learning models in computer vision for image recognition. Intelligent video analysis is generally limited by network bandwidth and cannot simultaneously aggregate and analyze multiple videos under limited bandwidth. Therefore, edge analysis within the local area network is generally adopted, that is, the intelligent video analysis system is deployed on the edge device to analyze the video stream output by the edge device, and then the analysis results are uploaded to the platform system. In the edge analysis process, the deep learning model is the determining factor in determining the accuracy of the final video analysis results. However, related technologies In this paper, the intelligent video analysis system and the deep learning model training system are two independent systems, essentially describing either the intelligent video analysis system or the deep learning model training system in a single context. The iteration of deep learning model capabilities is generally not considered in intelligent video analysis. Furthermore, the deep learning model training system generally does not consider the specific scenarios in which the deep learning model will be applied, or whether the model's recognition is correct or incorrect during application. Furthermore, even when iteration of deep learning model capabilities is considered, related technologies using deep learning methods to train deep learning models typically deploy the model in the intelligent video analysis system following the typical steps of data collection, labeling, model training, model evaluation, and model deployment. During the actual operation of the intelligent video analysis system, a series of model iterations are performed, including continuous sample collection and labeling, retraining the deep learning model, and re-deployment back to the intelligent video analysis system. From model training to model deployment and application, manual offline operations are generally required, reducing the efficiency of video image analysis. Furthermore, when selecting abnormal targets as new training samples to iteratively update the deep learning model, manual screenshots of the abnormal targets in the video images are required, resulting in ineffective iteration of the deep learning model. Based on this, the embodiments of the present invention provide a video image processing method and a target detection model training method. This technology can be applied to applications that require analysis of video images.
[0025] To facilitate understanding of this embodiment, a video image processing method disclosed in an embodiment of the present invention is first introduced. The method is applied to edge devices, such as Figure 1 As shown, the method includes the following steps:
[0026] Step S102: acquiring multiple frames of video images based on the original video stream; wherein each frame of the video image includes at least one target object;
[0027] The above-mentioned original video stream can be a video stream collected by an edge device, for example, the edge device can be a camera, etc.; the above-mentioned target object can be a person, a vehicle, a helmet, a shoe, etc.; in actual implementation, the original video stream collected by the edge device can be accessed, and the original video stream can be decoded to obtain multiple frames of video images, each frame of the video image usually includes one or more target objects.
[0028] Step S104: Input each frame of video image into a pre-trained target detection model, and output target detection results corresponding to each target object in each frame of video image; wherein the target detection results include: a target box corresponding to the target object and detection information; the detection information is used to indicate whether the target object has abnormal behavior;
[0029] The above-mentioned target detection model can be implemented through a variety of convolutional neural networks, such as residual networks, VGG networks, etc.; the above-mentioned target box can be a rectangular box used to represent the position and size of the target object; the above-mentioned abnormal behavior is usually related to the actual scene. For example, in the production scene, the abnormal behavior can be not wearing a safety helmet, not wearing safety gloves, etc.; in actual implementation, each frame of the video image obtained by the above decoding can be input into a pre-trained target detection model respectively, and each frame of the video image is detected by the target detection model to obtain the target box corresponding to each target object in each frame of the video image and the corresponding detection information; for example, a frame of the video image includes two production personnel, the first production personnel wears a full set of safety protection equipment, and the second production personnel does not wear a safety helmet. After detection by the target detection model, the target boxes corresponding to the two production personnel in the frame of the video image and their corresponding detection information can be obtained. For example, the detection information corresponding to the first production personnel can be "wearing qualified", and the detection information corresponding to the second production personnel can be "not wearing a safety helmet", etc.
[0030] Step S106: determining at least one abnormal image sequence based on the multiple target detection results; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior;
[0031] Each of the above-mentioned abnormal image sequences usually includes multiple consecutive frames of images; in actual implementation, after obtaining the target detection results corresponding to each target object in each frame of video image, if it is determined that one or more target objects have abnormal behavior based on these target detection results, then the abnormal image sequence corresponding to the target object can be generated based on the image area corresponding to the target frame of the same target object with abnormal behavior. If there are multiple different target objects with abnormal behavior, then a corresponding abnormal image sequence can be generated for each target object with abnormal behavior.
[0032] Step S108: for each abnormal image sequence, screen out a target abnormal image from the abnormal image sequence according to a preset optimal image determination method;
[0033] The above-mentioned optimal image determination method can be determined by a pre-set image evaluation index, and the image evaluation index can be set according to actual needs. For example, it can be the image average brightness value, image clarity, image size, image sharpness, etc., and an image evaluation index can be selected according to actual needs to obtain the above-mentioned optimal image determination method. For example, the optimal image determination method can be to select the image with the highest image average brightness value as the optimal image; or the above-mentioned optimal image determination method can be obtained by combining multiple image evaluation indicators. For example, the optimal image determination method can be the image with the highest value obtained after weighted calculation of the image average brightness value and image clarity as the optimal image, etc.; in actual implementation, after obtaining the above-mentioned one or more abnormal image sequences, the optimal target abnormal image can be selected from each abnormal image sequence according to the preset optimal image determination method.
[0034] Step S110 : If an error determination indication for the target abnormal image is received, the target abnormal image is sent to the server so as to update the weight parameters of the target detection model through the server.
[0035] In actual implementation, considering that during the training process of the target detection model, there may be missing samples related to the actual application scenarios in the training samples, which may lead to the problem of target detection model detection errors when the target detection model is actually applied. Therefore, for each abnormal image sequence, after screening out the target abnormal images in the abnormal image sequence, the screened out target abnormal images can be manually reviewed. If it is determined that the detection is wrong, the above-mentioned error judgment indication can be issued. For example, the detection information corresponding to the target object in the target abnormal image is "not wearing a safety helmet", but after manual review, it is determined that the target object in the target abnormal image is actually wearing a safety helmet, then the target abnormal image is a difficult sample for the target detection model. In this case, the target abnormal image can be sent to the server as a new sample, and the server can update the weight parameters of the target detection model based on the target abnormal image to achieve iterative optimization of the target detection model.
[0036] The above-mentioned video image processing method obtains multiple frames of video images based on the original video stream; wherein, each frame of the video image includes at least one target object; each frame of the video image is input into a pre-trained target detection model, and the target detection results corresponding to each target object in each frame of the video image are output; wherein, the target detection results include: a target box and detection information corresponding to the target object; the detection information is used to indicate whether the target object has abnormal behavior; based on multiple target detection results, at least one abnormal image sequence is determined; wherein, the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; for each abnormal image sequence, according to a preset optimal image determination method, a target abnormal image is screened out from the abnormal image sequence; if an error judgment indication for the target abnormal image is received, the target abnormal image is sent to the server to update the weight parameters of the target detection model through the server. In this method, the target abnormal image is screened according to a preset optimal image determination method. After receiving an error judgment indication for the target abnormal image, the weight parameters of the target detection model can be updated based on the target abnormal image to continue detecting the video image using the updated target detection model, thereby improving the iterative effect of the target detection model and improving the efficiency of video image analysis.
[0037] The embodiment of the present invention further provides another video image processing method, which is implemented on the basis of the method in the above embodiment, and includes the following steps:
[0038] Step 1: acquiring multiple frames of video images based on the original video stream; wherein each frame of the video image includes at least one target object;
[0039] Step 2: Input each frame of video image into a pre-trained target detection model and output the target detection results corresponding to each target object in each frame of video image. The target detection results include: the target box corresponding to the target object and detection information; the detection information is used to indicate whether the target object has abnormal behavior;
[0040] Step 3: Using the first frame of video image as the current video image, if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior, target tracking processing is performed on the abnormal target corresponding to the abnormal detection information;
[0041] The above-mentioned target tracking processing refers to the process of accurately locating and tracking the target in the next frame of video image based on the existing target position information in a continuous image sequence. In actual implementation, the first frame of video image can be used as the current video image, and the current video image may include one or more target objects. It is determined whether there is abnormal detection information indicating that the target object has first exhibited abnormal behavior in the target detection result corresponding to the current video image, that is, it is determined whether there are one or more target objects in the current video image that have abnormal behavior, and are detected for the first time in the current video image, and have not been detected in the video image before the current video image; if there is one or more abnormal detection information, such as "not wearing a helmet", etc., the target object corresponding to each abnormal detection information is determined as an abnormal target, and target tracking processing is performed on each abnormal target separately, that is, each abnormal target is located and tracked in other video images after the current video image.
[0042] Step 4: determining an abnormal image sequence based on image areas corresponding to target frames of abnormal targets in the current video image and other video images subsequent to the current video image;
[0043] For each abnormal target, among the other video images after the current video image, the abnormal target may only exist in a part of the video images. This part of the video images can be filtered out from the other video images after the current video image, and the image area corresponding to the target frame of the abnormal target in the current video image and this part of the video images can be used to form an abnormal image sequence corresponding to the abnormal target. For example, Figure 2 A schematic diagram of an abnormal image sequence is shown, where each image in the abnormal image sequence contains the same on-site worker.
[0044] Step 5: Using the next frame of video image as the new current video image, repeat the steps of determining if the target detection result corresponding to the current video image contains abnormality detection information indicating that the target object has first exhibited abnormal behavior, until the current video image is the last frame of video image.
[0045] The next frame of video image is used as the new current video image, and the above steps are repeated. After each frame of video image is executed according to the above steps, at least one abnormal image sequence can be obtained, and a unique ID (Identity document) can be set for each abnormal target in the abnormal image sequence.
[0046] Step 6: For each abnormal image sequence, for each abnormal image in the abnormal image sequence, obtain multiple image evaluation index data corresponding to the abnormal image, and perform weighted calculation on the multiple image evaluation index data according to a preset weight coefficient to obtain a weighted result corresponding to the abnormal image; the multiple image evaluation index data include: gradient richness data, image average brightness value, and number of pixels;
[0047] The aforementioned gradient richness data can be understood as the summary of the gradient amplitudes of all pixels in the abnormal image, reflecting the overall edge strength and detail richness of the abnormal image. A high gradient richness generally indicates that the abnormal image has more edges and details, while a low gradient richness indicates that the abnormal image is relatively flat and lacks distinct edges. The aforementioned image average brightness value refers to the average brightness of all pixels in the abnormal image, reflecting the overall brightness level of the abnormal image. The aforementioned number of pixels generally directly affects the clarity and detail of the abnormal image. Generally, the greater the number of pixels, the higher the resolution of the abnormal image, the richer the details, and the clearer the image. In actual implementation, for each abnormal image sequence, for each abnormal image in the abnormal image sequence, multiple image evaluation index data corresponding to the abnormal image can be calculated. In this embodiment, the multiple image evaluation index data specifically include: gradient richness data, image average brightness value, and number of pixels. Each image evaluation index data can be pre-configured with a corresponding weight coefficient as needed. Through weighted calculation, a weighted result corresponding to the abnormal image can be obtained. The weighted result corresponding to each abnormal image in the abnormal image sequence may be the same or different.
[0048] The above gradient richness data is determined by following steps 60 and 61:
[0049] Step 60, calculating the gradient score of each pixel in the abnormal image according to a preset gradient method;
[0050] Step 61: sum the gradient scores of each pixel to obtain gradient richness data.
[0051] The above gradient method calculation formula can be as follows:
[0052]
[0053] Where SR(x, y) represents the gradient score of pixel (x, y); f(x, y) represents the pixel value of pixel (x, y); w i，j represents the weight; m and n represent the size of the convolution kernel.
[0054] For each abnormal image in the abnormal image sequence, the gradient score of each pixel in the abnormal image can be calculated according to the above formula. Generally, a larger gradient score indicates a higher image sharpness, while a smaller gradient score indicates a lower image sharpness. By summing the gradient scores of each pixel, we can obtain the gradient richness data corresponding to the abnormal image.
[0055] When it is necessary to calculate the average brightness value of the image corresponding to the abnormal image, it is usually necessary to first determine whether the abnormal image needs to be converted based on the color model of the abnormal image. Specifically, if the abnormal image is a color image, it is usually necessary to convert the abnormal image into a grayscale image first. If the abnormal image itself is a grayscale image, no conversion is required. Then, the average brightness value of the abnormal image is calculated according to the following formula:
[0056]
[0057] Wherein, avegray represents the average brightness value of the image; w represents the image width of the abnormal image; h represents the image height of the abnormal image; i represents the i-th pixel in the image width direction; j represents the j-th pixel in the image height direction.
[0058] The above number of pixels is the product of the image width and image height of the abnormal image. Specifically, the number of pixels of the abnormal image can be calculated using the following formula:
[0059] Pixelnum = w * h;
[0060] Wherein, Pixelnum represents the number of pixels; w represents the image width of the abnormal image; and h represents the image height of the abnormal image.
[0061] Step 7: Determine the target abnormal image as the abnormal image corresponding to the weighted result with the largest value among the multiple weighted results;
[0062] Image evaluation index data such as gradient richness data, image average brightness value, and number of pixels are meaningless for comparison of different pictures and different targets, but for relative comparison of the same target object from far to near or from near to far in the same scene, the best one can be selected. In this embodiment, based on the abnormal image sequence corresponding to the same abnormal target, a weighted calculation of the three dimensions of gradient richness data, image average brightness value, and number of pixels is performed on each abnormal image, and the weighted results corresponding to each abnormal image are sorted in descending order, and the abnormal image with the highest ranking is selected as the target abnormal image; if there are multiple weighted results with the largest values, the abnormal image corresponding to any one of the weighted results can be selected as the target abnormal image.
[0063] For example, the weight coefficients corresponding to the gradient richness data, the average brightness value of the image, and the number of pixels are 0.5, 0.3, and 0.2, respectively. For any abnormal image in the abnormal image sequence, the weighted result corresponding to the abnormal image can be calculated as follows:
[0064] Sort=0.5*Sort_grad+0.3*Sort_avegray+0.2*Sort_pixelnum;
[0065] Among them, Sort represents the weighted result of the abnormal image; Sort_grad represents the gradient richness data of the abnormal image; Sort_avegray represents the average brightness value of the abnormal image; Sort_pixelnum represents the number of pixels in the abnormal image.
[0066] Step eight, screening out a target abnormal image from the abnormal image sequence;
[0067] Step nine, generating warning information for the target abnormal image;
[0068] Step 10: If an error determination indication is received for the alarm information, the target abnormal image is sent to the server to update the reflow sample set in the server, and the weight parameters of the target detection model are updated based on the reflow sample set by the server.
[0069] For each abnormal image sequence, after determining the target abnormal image in the abnormal image sequence, the target abnormal image can be screened out and an alarm message can be generated to prompt the business personnel to manually review the target abnormal image indicated by the alarm message. If the detection is determined to be correct, the business can be archived; if it is determined that the detection is wrong, the above-mentioned error judgment indication can be issued, and the target abnormal image can be sent to the server, and the server saves the target abnormal image to a preset reflow sample set; the reflow sample set can contain multiple different target abnormal images; the weight parameters of the target detection model can be updated based on each target abnormal image in the reflow sample set; for example, at regular intervals, each target object in each target abnormal image in the reflow sample set can be manually labeled with a standard box and a standard detection result, and the weight parameters of the target detection model can be fine-tuned as a new training sample, and the fine-tuned target detection model can be sent to the intelligent video analysis system to replace the original target detection model to continue analyzing the video image.
[0070] In the above-mentioned video image processing method, after the target detection model is used to output the target detection results, the abnormal image sequence can be determined by target tracking processing, and then the target abnormal images can be screened out and alarms can be issued based on multiple image evaluation index data. After manual review, the target abnormal images that are falsely alarmed can be sent back to the server to update the weight parameters of the target detection model, ensuring that the samples participating in the iterative training of the model are high-quality optimal negative samples, which can greatly iterate the performance of the model and thereby improve the efficiency of video image analysis.
[0071] The embodiment of the present invention provides a method for training a target detection model, which is applied to a server, such as Figure 3 As shown, the method includes the following steps:
[0072] Step S302: determining a sample image based on a preset sample set; wherein the sample set includes: a sample subset collected by an edge device; the sample image includes at least one sample target, each sample target being marked with a standard frame and a standard detection result;
[0073] The above-mentioned sample targets can be: people, vehicles, helmets, shoes, etc.; the above-mentioned standard box can be understood as a pre-marked rectangular box used to indicate the position and size of the sample target; the above-mentioned standard detection result is a pre-marked result used to indicate whether the sample target has abnormal behavior: in actual implementation, a large number of general data sets and the above-mentioned sample subsets can be obtained to constitute the above-mentioned sample set. Since the sample subset is collected by the edge device that collects the original video stream, the sample image in the sample subset usually has the same specific scene as the original video stream; in specific implementation, the sample subset can be collected by the edge device in a specific scenario. For example, the sample video stream can be collected, and the sample subset can be obtained from the sample video stream by frame extraction or preset large model screening. For the sample subset, the sample target in each sample image is marked with the corresponding standard box and standard detection result.
[0074] Step S304: input the sample image into the initial model, so that the initial model outputs the prediction box and the prediction detection result corresponding to the sample image;
[0075] The above-mentioned initial model can be implemented through a variety of convolutional neural networks, such as residual networks, VGG networks, etc.; in actual implementation, the above-mentioned sample image can be input into the initial model, and the initial model can be used to predict the prediction box and predicted detection result corresponding to each sample target in the sample image; the prediction box can be understood as the predicted rectangular box representing the position and size of the sample target; the above-mentioned predicted detection result can be understood as the predicted result used to indicate whether the sample target has abnormal behavior; for the same sample target, the prediction box corresponding to the sample target may be the same or different from the corresponding standard box, and the prediction detection result corresponding to the sample target may be the same or different from the standard detection result.
[0076] In step S306, a loss value is determined based on the standard box, the predicted box, the standard detection result, and the predicted detection result, so as to update the initial model based on the loss value until the loss value converges to obtain a target detection model.
[0077] In actual implementation, the above loss value is a value related to the distance between the standard box and the predicted box, as well as the distance between the standard detection result and the predicted detection result. The weight parameters of the initial model can be continuously adjusted according to the loss value until the loss value converges to obtain a trained target detection model.
[0078] The above steps of updating the initial model based on the loss value until the loss value converges to obtain the target detection model can be determined by the following steps A and B:
[0079] Step A: updating the initial model based on the loss value until the loss value converges to obtain a first detection model;
[0080] Step B: convert the first detection model to obtain a target detection model.
[0081] In actual implementation, after executing the above training process until the loss value converges, a trained first detection model can be obtained first. In order to adapt to the intelligent video analysis system deployed on the edge device, before sending the first detection model to the intelligent video analysis system for use, it is usually necessary to convert and optimize the first detection model to improve the detection effect and detection efficiency for the intelligent video analysis; specifically, the training framework used by the first detection model in the training process may be different from the model framework that can be used by the intelligent video analysis system, then the first detection model needs to be converted into a model framework that can be used by the intelligent video analysis system. For example, the first detection model uses If the training framework is pytorch or paddle, and the intelligent video analysis system can use the onnx model framework, the first detection model needs to be converted into the onnx model framework and sent to the intelligent video analysis system for use; in addition, in order to further improve the detection efficiency, the weight parameters of the first detection model can be uniformly converted into integer form. For example, the weight parameter 1.2×1.5 can be converted to 12×15÷100, etc. By performing model framework conversion and weight parameter form conversion on the first detection model, the converted target detection model can be obtained, and the target detection model can be sent to the intelligent video analysis system for detection and processing of video images.
[0082] In the training method of the above-mentioned target detection model, the sample set includes a sample subset collected by edge devices, so the sample images in the sample subset are more in line with the specific scenarios of actual applications. The initial model is trained based on the sample images determined by the sample set, which can improve the iterative effect of the obtained target detection model, thereby improving the efficiency of video image analysis in actual applications.
[0083] An embodiment of the present invention provides a video image processing system, the system comprising: an edge device and a server;
[0084] The server is used to adopt the target detection model training method of any of the above embodiments to obtain the target detection model, and send the target detection model to the edge device; the edge device is used to adopt the video image processing method of any of the above embodiments based on the target detection model to screen out target abnormal images, and if an error judgment indication for the target abnormal image is received, the target abnormal image is sent to the server; the server is used to update the reflow sample set according to the target abnormal image, and update the weight parameters of the target detection model based on the reflow sample set. For details, please refer to the relevant description in the above embodiments, which will not be repeated here.
[0085] For easier understanding, see Figure 4A schematic diagram of a video image processing system is shown, which includes: an intelligent video analysis system running in an edge device and a model training system running in a server. The model training system includes sample collection, sample organization, sample labeling, model training and model evaluation steps, wherein the collected sample set includes a general data set and offline scene data (corresponding to the above-mentioned sample subset) collected by the edge device (such as a camera) corresponding to the application scenario. After the model evaluation is completed, a first detection model can be obtained, and the first detection model can be sent to the intelligent video analysis system. It is usually necessary to first perform model conversion / optimization processing on the first detection model to obtain a target detection model. The intelligent video analysis system can use the target detection model to analyze the video image.
[0086] The intelligent video analysis system can decode the received original video stream to obtain multiple frames of video images, use the target detection model to perform image target detection on the multiple frames of video images, perform target tracking processing on the detected abnormal targets to obtain an abnormal image sequence, select the optimal target abnormal image from the abnormal image sequence, perform alarm processing by executing the alarm logic, and then conduct an alarm review. Specifically, the target abnormal image can be manually reviewed, and the target abnormal image that is detected incorrectly is returned to the model training system. After re-labeling, the target detection model is fine-tuned and iteratively optimized.
[0087] The entire process of the above-mentioned video image processing is automatically completed in the system, which not only lowers the threshold for model training, but also improves the efficiency of model iteration and implementation. It is a video image processing system with self-iteration and repair capabilities.
[0088] In actual applications, one edge device can correspond to one intelligent video analysis system, and one intelligent analysis system can correspond to one first detection model; or multiple intelligent video analysis systems can correspond to the same first detection model. In this case, when the first detection model needs to be sent, it can be sent to the corresponding intelligent video analysis system based on the platform information of the edge device, where the platform information may include: device name, number and other information of the edge device.
[0089] Before the application goes online, the above-mentioned video image processing method collects a subset of samples collected by edge devices (such as cameras) in specific scenarios, combines them with a general data set as a sample set, and trains an initial model after sample organization and labeling in the model training system. After the training is completed, a target detection model is obtained. After the target detection model is deployed to the intelligent video analysis system, the target detection model can be used for video image analysis.
[0090] After the target detection model is deployed to the intelligent video analysis system, the offline trained target detection model is run for a period of time in various scenarios to obtain alarm results for abnormal target images. After the business personnel review the alarm results, they can periodically (for example, in the early stage of the target detection model being deployed to the intelligent video analysis system, it can be returned once a week, and in the later stage, it can be returned once a month) return the false alarm images to the model training system for training. If there is no deviation in the training evaluation, the fine-tuned target detection model will be automatically sent to the intelligent video analysis system.
[0091] The above-mentioned video image processing method uses the optimal image judgment algorithm in the tracking trajectory to screen the optimal image for each abnormal target alarm. The optimal image judgment is obtained by relatively comparing abnormal images of the same abnormal target, ensuring that the samples participating in the iterative training of the model are high-quality optimal negative samples, which can greatly iterate the performance of the model.
[0092] The training of the target detection model is divided into two processes: offline training and online training. In the offline model training stage, general datasets and sample subsets are used for training; in the online model training stage, abnormal target images with detection errors are used to perform fine-tuning training based on the offline model.
[0093] This solution establishes a system of intelligent video analysis data closed loop, model closed loop and application closed loop. It is a security early warning system that integrates intelligent video analysis and model training. The application goal of this system is to strongly bind the model training work to the early warning system of intelligent video analysis. The iteration of the target detection model is inseparable from the increase and optimization of data. During the application process, target abnormality images that improve the model effect and business can be precipitated. The target detection model can be fine-tuned based on the target abnormality images to iteratively optimize the target detection model, so that the training iteration of the model can directly affect the business. In this solution, through the direct participation of business personnel in the improvement of the model, the target abnormality images with detection errors are returned to the model training system, which can greatly enhance the model's capabilities.
[0094] The embodiment of the present invention provides a video image processing device, which is arranged on an edge device, such as Figure 5As shown, the device includes: an acquisition module 40, which is used to acquire multiple frames of video images based on the original video stream; wherein each frame of the video image includes at least one target object; a first output module 41, which is used to input each frame of the video image into a pre-trained target detection model, and output the target detection results corresponding to each target object in each frame of the video image; wherein the target detection results include: a target box and detection information corresponding to the target object; the detection information is used to indicate whether the target object has abnormal behavior; a first determination module 42, which is used to determine at least one abnormal image sequence based on multiple target detection results; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; a screening module 43, which is used to screen out target abnormal images from the abnormal image sequence according to a preset optimal image determination method for each abnormal image sequence; an updating module 44, which is used to send the target abnormal image to a server if an error judgment indication for the target abnormal image is received, so as to update the weight parameters of the target detection model through the server.
[0095] In the above-mentioned video image processing device, the target abnormal image is screened according to the preset optimal image determination method. After receiving the error judgment indication for the target abnormal image, the weight parameters of the target detection model can be updated based on the target abnormal image to continue to detect the video image using the updated target detection model, thereby improving the iterative effect of the target detection model and improving the efficiency of video image analysis.
[0096] Furthermore, the first determination module is also used to: take the first frame of video image as the current video image, and if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior, perform target tracking processing on the abnormal target corresponding to the abnormal detection information; determine the abnormal image sequence based on the image area corresponding to the target box of the abnormal target in the current video image and other video images after the current video image; take the next frame of video image as the new current video image, and repeat the steps of if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior, until the current video image is the last frame of video image.
[0097] Furthermore, the screening module is also used to: for each abnormal image sequence, for each abnormal image in the abnormal image sequence, obtain multiple image evaluation index data corresponding to the abnormal image, perform weighted calculation on the multiple image evaluation index data according to a preset weight coefficient, and obtain a weighted result corresponding to the abnormal image; the multiple image evaluation index data include: gradient richness data, image average brightness value, and number of pixels; determine the target abnormal image as the abnormal image corresponding to the weighted result with the largest value among the multiple weighted results; and screen out the target abnormal image from the abnormal image sequence.
[0098] Furthermore, the update module is also used to: generate an alarm message for the target abnormal image; if an error judgment indication for the alarm message is received, send the target abnormal image to the server to update the reflow sample set in the server, and update the weight parameters of the target detection model based on the reflow sample set through the server.
[0099] Furthermore, the device also includes a gradient richness data determination module, which is used to: calculate the gradient score of each pixel in the abnormal image according to a preset gradient method; and add the gradient score of each pixel to obtain gradient richness data.
[0100] The video image processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned video image processing method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the video image processing device, reference can be made to the corresponding content in the aforementioned video image processing method embodiment.
[0101] The embodiment of the present invention provides a training device for a target detection model, which is set on a server, such as Figure 6 As shown, the device includes: a second determination module 60, which is used to determine a sample image based on a preset sample set; wherein the sample set includes: a sample subset collected by an edge device; the sample image includes at least one sample target, and each sample target is marked with a standard box and a standard detection result; a second output module 61, which is used to input the sample image into the initial model to output the predicted box and predicted detection result corresponding to the sample image through the initial model; a third determination module 62, which is used to determine the loss value based on the standard box, the predicted box, the standard detection result and the predicted detection result, so as to update the initial model based on the loss value until the loss value converges to obtain the target detection model.
[0102] The training device of the above-mentioned target detection model includes a sample subset collected by the edge device in the sample set, so that the sample images in the sample subset are more in line with the specific scenarios of the actual application. The initial model is trained based on the sample images determined by the sample set, which can improve the iterative effect of the obtained target detection model, thereby improving the efficiency of video image analysis in practical applications.
[0103] Furthermore, the third determination module is also used to: update the initial model based on the loss value until the loss value converges to obtain a first detection model; and convert the first detection model to obtain a target detection model.
[0104] The training device for the target detection model provided in the above-mentioned embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned embodiment of the training method for the target detection model. For the sake of brief description, for matters not mentioned in the embodiment of the training device for the target detection model, reference may be made to the corresponding content in the aforementioned embodiment of the training method for the target detection model.
[0105] The embodiment of the present invention further provides an electronic device, see Figure 7 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned video image processing method or target detection model training method.
[0106] Furthermore, Figure 7 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .
[0107] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.
[0108] The processor 130 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 130 or by software instructions. The above processor 130 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 131, and processor 130 reads information in memory 131 and, in conjunction with its hardware, completes the steps of the method of the aforementioned embodiment.
[0109] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned video image processing method or target detection model training method. The specific implementation can be found in the method embodiment and will not be repeated here.
[0110] The computer program products of the video image processing method and the target detection model training method provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.
[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A video image processing method, characterized in that: Applied to an edge device, the method includes: Acquire multiple frames of video images based on the original video stream; wherein each frame of the video image includes at least one target object; Input each frame of video image into a pre-trained target detection model, and output the target detection results corresponding to each target object in each frame of the video image; wherein the target detection results include: a target box corresponding to the target object and detection information; the detection information is used to indicate whether the target object has abnormal behavior; Determining at least one abnormal image sequence based on the plurality of target detection results; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; For each abnormal image sequence, a target abnormal image is screened out from the abnormal image sequence according to a preset optimal image determination method; If an error determination indication is received for the target abnormal image, the target abnormal image is sent to a server so as to update weight parameters of the target detection model through the server.
2. The method according to claim 1, characterized in that The step of determining at least one abnormal image sequence based on the plurality of target detection results comprises: The first frame of the video image is used as the current video image, and if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior, target tracking processing is performed on the abnormal target corresponding to the abnormal detection information; determining an abnormal image sequence according to image regions corresponding to target frames of the abnormal targets in the current video image and other video images subsequent to the current video image; The next frame of video image is used as the new current video image, and the step of determining if the target detection result corresponding to the current video image contains abnormal detection information indicating that the target object has first exhibited abnormal behavior is repeated until the current video image is the last frame of video image.
3. The method according to claim 1, characterized in that For each abnormal image sequence, the step of screening out a target abnormal image from the abnormal image sequence according to a preset optimal image determination method includes: For each abnormal image sequence, for each abnormal image in the abnormal image sequence, a plurality of image evaluation index data corresponding to the abnormal image is obtained, and a weighted calculation is performed on the plurality of image evaluation index data according to a preset weight coefficient to obtain a weighted result corresponding to the abnormal image; the plurality of image evaluation index data includes: gradient richness data, image average brightness value, and number of pixels; The abnormal image corresponding to the weighted result with the largest value among the multiple weighted results is determined as the target abnormal image; The target abnormal image is screened out from the abnormal image sequence.
4. The method according to claim 1, wherein If an error determination indication is received for the target abnormal image, the step of sending the target abnormal image to a server so as to update the weight parameters of the target detection model through the server includes: generating warning information for the target abnormal image; If an error determination indication is received for the warning information, the target abnormal image is sent to the server to update the reflow sample set in the server, and the weight parameters of the target detection model are updated based on the reflow sample set by the server.
5. The method according to claim 3, characterized in that The gradient richness data is determined by: Calculate the gradient score of each pixel in the abnormal image according to the preset gradient method; The gradient score of each pixel is summed to obtain the gradient richness data.
6. A method for training a target detection model, characterized in that: Applied to a server, the method includes: Determine a sample image based on a preset sample set; wherein the sample set includes: a sample subset collected by an edge device; the sample image includes at least one sample target, each of the sample targets is marked with a standard frame and a standard detection result; Inputting the sample image into the initial model so as to output the prediction box and the prediction detection result corresponding to the sample image through the initial model; Based on the standard box, the predicted box, the standard detection result and the predicted detection result, a loss value is determined to update the initial model based on the loss value until the loss value converges to obtain the target detection model.
7. The method according to claim 6, characterized in that The step of updating the initial model based on the loss value until the loss value converges to obtain the target detection model includes: Updating the initial model based on the loss value until the loss value converges to obtain a first detection model; The first detection model is converted to obtain the target detection model.
8. A video image processing system, characterized in that: The system includes: an edge device and a server; The server is configured to adopt the target detection model training method according to any one of claims 6-7 to obtain a target detection model, and send the target detection model to the edge device; The edge device is configured to screen out abnormal target images based on the target detection model and adopt the video image processing method according to any one of claims 1 to 5, and if an error determination indication is received for the abnormal target image, send the abnormal target image to the server; The server is configured to update a reflow sample set according to the target abnormal image, and update a weight parameter of the target detection model based on the reflow sample set.
9. A video image processing device, characterized in that: Set on the edge device, the device includes: An acquisition module, configured to acquire multiple frames of video images based on an original video stream, wherein each frame of the video image includes at least one target object; A first output module is configured to input each frame of video image into a pre-trained target detection model and output a target detection result corresponding to each target object in each frame of the video image; wherein the target detection result includes: a target frame corresponding to the target object and detection information; the detection information is used to indicate whether the target object has abnormal behavior; A first determination module is configured to determine at least one abnormal image sequence based on the plurality of target detection results; wherein the same abnormal image sequence corresponds to the same target object, and the detection information corresponding to the target object in each abnormal image sequence indicates that the target object has abnormal behavior; a screening module, configured to screen out target abnormal images from each abnormal image sequence according to a preset optimal image determination method; An updating module is configured to send the target abnormal image to a server if an error determination indication is received for the target abnormal image, so as to update the weight parameters of the target detection model through the server.
10. A training device for a target detection model, characterized in that: Set on the server, the device includes: A second determination module is configured to determine a sample image based on a preset sample set; wherein the sample set includes: a sample subset collected by an edge device; the sample image includes at least one sample target, each of the sample targets being annotated with a standard frame and a standard detection result; A second output module is used to input the sample image into the initial model to output the prediction box and the prediction detection result corresponding to the sample image through the initial model; The third determination module is used to determine the loss value based on the standard box, the predicted box, the standard detection result and the predicted detection result, so as to update the initial model based on the loss value until the loss value converges to obtain the target detection model.