Model updating method and device, equipment, storage medium

Abstract

The application discloses a model updating method and device, equipment and a storage medium. The method comprises the following steps: receiving a first fault image sent by a first electronic device; the first fault image is detected from a monitoring video by a first fault detection model; at least one first fault image is used to train a stored first fault detection model to obtain a first target model; and model parameters of the first target model are sent to the first electronic device to update the first fault detection model. Thus, the problem of difficult model iteration update caused by limited computing power of the first electronic device can be solved.

Description

Technical Field

[0001] This application relates to video surveillance technology, including but not limited to model update methods, devices, equipment, and storage media. Background Technology

[0002] Video surveillance is a crucial component of security systems, widely used in homes, industrial parks, cities, educational institutions, and industries. With the explosive growth in the scale of surveillance video, malfunctions such as screen tearing, obstruction, and pixelation in surveillance equipment are inevitable. Therefore, fault detection for surveillance systems is of paramount importance. As the volume of surveillance video data continues to increase, fault detection of surveillance equipment becomes more complex. Relying solely on human resources for troubleshooting is no longer sufficient to meet practical needs, necessitating the development of intelligent fault detection methods for surveillance video systems.

[0003] The deep learning-based video quality diagnosis method analyzes the faults in surveillance video images, establishes the correspondence between front-end acquisition faults and video image quality, collects and creates a comprehensive dataset of surveillance video faults, and uses deep learning models to extract more advanced and comprehensive features of abnormal videos, thereby training a more accurate fault detection model with stronger universality in practical applications.

[0004] However, while deploying fault detection models on monitoring equipment can achieve real-time detection of monitoring faults, it presents the problem of high difficulty in model iteration. Summary of the Invention

[0005] In view of this, the model update method, apparatus, device, and storage medium provided in this application can solve the problem of difficulty in iterative model update caused by the limited computing power of the first electronic device.

[0006] According to one aspect of the embodiments of this application, a model update method is provided, comprising: receiving a first fault image sent by a first electronic device; wherein the first fault image is detected by the first electronic device from a monitoring video by the first electronic device through a first fault detection model; training at least one stored first fault image based on at least one of the first fault images to obtain a first target model; and sending the model parameters of the first target model to the first electronic device for the first electronic device to update the first fault detection model; thereby solving the problem of difficulty in model iterative update caused by the limited computing power of the first electronic device.

[0007] According to one aspect of the embodiments of this application, another model update method is provided, comprising: detecting fault images present in a surveillance video using a first fault detection model to obtain a first fault image; sending the first fault image to a second electronic device; wherein the first fault image is used by the second electronic device to train a stored first fault detection model; receiving model parameters of a first target model sent by the second electronic device; the first target model is obtained by the second electronic device training the first fault detection model based on at least one of the first fault images; and updating the first fault detection model based on the model parameters of the first target model.

[0008] According to one aspect of the embodiments of this application, a model update apparatus is provided, comprising: a first receiving module, configured to receive a first fault image sent by a first electronic device; wherein the first fault image is detected by the first electronic device from a monitoring video using a first fault detection model; a training module, configured to train at least one stored first fault detection model based on at least one of the first fault images to obtain a first target model; and a first sending module, configured to send the model parameters of the first target model to the first electronic device for the first electronic device to update the first fault detection model.

[0009] According to one aspect of the embodiments of this application, a model update apparatus is provided, comprising: a fault detection module, configured to detect fault images present in a surveillance video using a first fault detection model to obtain a first fault image; a second sending module, configured to send the first fault image to a second electronic device; wherein the first fault image is used by the second electronic device to train a stored first fault detection model; a second receiving module, configured to receive model parameters of a first target model sent by the second electronic device; the first target model is obtained by the second electronic device training the first fault detection model based on at least one of the first fault images; and a parameter update module, configured to update the first fault detection model based on the model parameters of the first target model.

[0010] The electronic device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0011] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method described in this application embodiment.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0015] Figure 1 A schematic diagram illustrating the implementation flow of the model update method provided in this application embodiment;

[0016] Figure 2 A schematic diagram illustrating the implementation flow of another model update method provided in this application embodiment;

[0017] Figure 3 A schematic diagram illustrating the implementation flow of another model update method provided in an embodiment of this application;

[0018] Figure 4 A schematic diagram illustrating the implementation flow of the method for determining the target type of the first fault image provided in this application embodiment;

[0019] Figure 5 This is a schematic diagram of the structure of the second fault detection model provided in the embodiments of this application;

[0020] Figure 6 This is a schematic diagram illustrating the implementation process of the labeling method provided in the embodiments of this application;

[0021] Figure 7 A schematic diagram illustrating the training method implementation flow of the second fault detection model provided in this application embodiment;

[0022] Figure 8 A schematic diagram of a cloud-based collaborative intelligent fault detection system for surveillance videos provided in this application embodiment;

[0023] Figure 9 This is a schematic diagram of the structure of the model update device provided in the embodiments of this application;

[0024] Figure 10 This is a schematic diagram of another model update device provided in an embodiment of this application;

[0025] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] It should be noted that the terms "first, second, third" used in the embodiments of this application do not represent a specific order of objects. It is understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0030] This application provides a model update method. Figure 1 This is a schematic diagram illustrating the implementation process of the model update method provided in the embodiments of this application, as shown below. Figure 1 As shown, the method may include the following steps 101 to 103:

[0031] Step 101: The second electronic device receives a first fault image sent by the first electronic device; wherein the first fault image is detected by the first electronic device from the monitoring video through a first fault detection model.

[0032] In this application, there is no limitation on the types of the first electronic device and the second electronic device, which can be various. For example, the first electronic device is a monitoring device, which may include a camera module; the monitoring device may also not include a camera module, and the camera module is independent of the monitoring device. In this case, the monitoring device can transmit the monitoring video to the monitoring device via wireless or wired communication, so that the monitoring device can detect whether there is a fault image in the monitoring video through the configured first fault detection model.

[0033] The second electronic device can be any type of device, such as a personal computer, a laptop, a single server, or a cluster of servers.

[0034] A faulty image refers to an image that does not meet quality requirements due to hardware or software malfunctions in the device or camera module. Examples of faulty images include blurry images, overly bright images, overly dark images, images with color casts, mosaic images, noisy images, images with camera obstruction, striped images, or images captured due to abnormal camera angles.

[0035] In this application, there are no restrictions on the structure of the first fault detection model. It can be any machine learning model that can identify fault images in surveillance videos, such as convolutional neural networks, linear regression, logistic regression, decision trees, random decision trees, or support vector machines.

[0036] Step 102: The second electronic device trains at least one of the first fault images on at least one stored first fault detection model to obtain a first target model.

[0037] The phrase "at least the first fault detection model is trained" includes: the second electronic device is trained only on the first fault detection model, or the second electronic device is trained on the second fault detection model to which the first fault detection model is located, i.e., the first fault detection model is a branch of the second fault detection model.

[0038] In some embodiments, the second electronic device is configured with a cloud platform, and the second electronic device performs step 102 through the running cloud platform. In some embodiments, the second electronic device can implement step 102 through steps 302 to 304 of the following embodiments, which will not be described again here.

[0039] In this application, the second electronic device can update the model parameters of the first fault detection model based on each first fault image it receives; the second electronic device can also train the first fault detection model based on these images after collecting a certain number of first fault images, that is, update the model parameters.

[0040] Step 103: The second electronic device sends the model parameters of the first target model to the first electronic device so that the first electronic device can update the first fault detection model.

[0041] In this embodiment, the second electronic device receives a first fault image sent by the first electronic device, and trains a stored first fault detection model based on at least one of the first fault images to obtain a first target model. Then, the model parameters of the first target model are sent to the first electronic device so that the first electronic device can update the first fault detection model it uses. In this way, the computationally expensive model training work is handed over to the second electronic device, thereby solving the problem of difficulty in model iteration and updating caused by the limited computing power of the first electronic device.

[0042] This application provides another model update method. Figure 2 This is a schematic diagram illustrating the implementation process of the model update method provided in the embodiments of this application, as shown below. Figure 2 As shown, the method includes steps 201 to 207:

[0043] Step 201: The first electronic device detects fault images in the monitoring video through the first fault detection model and obtains the first fault image.

[0044] In some embodiments, the first electronic device can adjust the size of the video frame of the monitoring video through the preprocessing layer of the first fault detection model to obtain a video frame with a first size; through the first sub-model of the first fault detection model, determine the fourth probability of the video frame with the first size belonging to each preset type; and based on the fourth probability of each preset type, determine whether the video frame is a fault image.

[0045] In this application, there are no restrictions on the preset types, which can be set according to the actual situation. For example, preset types include, but are not limited to: normal images, blurry images, overly bright images, overly dark images, color-distorted images, mosaic images, noisy images, images obstructed by the camera, striped images, or abnormal images such as images taken due to abnormal camera angles.

[0046] In this application, the size of the first dimension is not limited and can be any size. For example, the first dimension is 224×224, that is, the video frame is converted into a small image with a resolution of 224×224 through the preprocessing layer. In some embodiments, in order to avoid being limited by the computing power of the first electronic device, the first sub-model is configured as a lightweight convolutional neural network.

[0047] Step 202: The first electronic device sends the first fault image to the second electronic device; wherein the first fault image is used by the second electronic device to train the stored first fault detection model.

[0048] In this application, the first electronic device may upload each first fault image it detects to the second electronic device; or, the first electronic device may transmit these images to the second electronic device after collecting a certain number of first fault images.

[0049] Step 203: The second electronic device receives the first fault image;

[0050] Step 204: The second electronic device trains at least one of the first fault images on the stored first fault detection model to obtain a first target model.

[0051] Step 205: The second electronic device sends the model parameters of the first target model to the first electronic device so that the first electronic device can update the first fault detection model.

[0052] Step 206: The first electronic device receives the model parameters of the first target model sent by the second electronic device;

[0053] Step 207: The first electronic device updates the first fault detection model based on the model parameters of the first target model.

[0054] In some embodiments, the first electronic device replaces the model parameters of the first fault detection model with the model parameters of the first target model.

[0055] Understandably, in video surveillance systems, the computing power of deployed edge devices (i.e., the first electronic device) varies, and the computing power of these devices is often limited. Therefore, in this embodiment, the first fault detection model on the first electronic device side is updated and iterated through cloud collaboration; that is, the second electronic device sends the model parameters of the trained first target model to the first electronic device, so that the first electronic device can replace its own first fault detection model parameters with them, thereby updating the model parameters; in this way, the problem of difficulty in iterating model parameters caused by the limited computing power of the first electronic device is solved.

[0056] This application embodiment further provides a model update method, which is applied to a second electronic device. Figure 3 This is a schematic diagram illustrating the implementation process of the model update method provided in the embodiments of this application, as shown below. Figure 3 As shown, the method includes steps 301 to 305:

[0057] Step 301: The second electronic device receives a first fault image sent by the first electronic device; wherein the first fault image is detected by the first electronic device from the monitoring video through a first fault detection model.

[0058] In some embodiments, the first electronic device includes a monitoring device, and the method is applied to a second electronic device configured to run a cloud platform for training a stored first fault detection model based on at least one of the first fault images.

[0059] Step 302: The second electronic device performs fault detection on the first fault image using the stored second fault detection model to obtain the target type of the first fault image; wherein, the first fault detection model is a branch of the second fault detection model;

[0060] Step 303: The second electronic device labels the first fault image based on the target type of the first fault image;

[0061] Step 304: The second electronic device trains the second fault detection model based on at least one first fault image and the labels of each of the first fault images to obtain a second target model; wherein the second target model includes the first target model;

[0062] Step 305: The second electronic device sends the model parameters of the first target model to the first electronic device so that the first electronic device can update the first fault detection model.

[0063] The following provides a detailed description of further implementation methods and key terms for some of the above steps.

[0064] In step 302, the second electronic device performs fault detection on the first fault image using the stored second fault detection model to obtain the target type of the first fault image; wherein, the first fault detection model is a branch of the second fault detection model.

[0065] It should be noted that the detection accuracy of the second fault detection model is higher than that of the first fault detection model. To reduce false detections by the first fault detection model on the first electronic device side, the first fault image needs to be re-detected to reconfirm its type, thereby improving the detection accuracy of the first fault image.

[0066] In some embodiments, the second fault detection model includes a third fault detection model, a first fault detection model, a fusion layer, and an output layer, wherein the third fault detection model has a different structure from the first fault detection model. Further, the depth of the third fault detection model can be greater than, less than, or equal to, the depth of the first fault detection model. The third fault detection model may include one or more neural network structures; similarly, the first fault detection model may also include one or more neural network structures.

[0067] In some embodiments, such as Figure 4 As shown, the second electronic device can implement step 302 through the following steps 3021 to 3024:

[0068] Step 3021: Process the first fault image using the first fault detection model to obtain the first probability that the first fault image belongs to each preset type.

[0069] In some embodiments, the size of the first fault image is adjusted by the preprocessing layer of the first fault detection model to obtain a first fault image with a first size; the first probability of the first fault image with the first size belonging to each of the preset types is determined by the first sub-model of the first fault detection model.

[0070] Understandably, the preprocessing layer is the resize layer, responsible for converting the first fault image into a first fault image with a first size, i.e., the image size supported by the first sub-model. There is no restriction on the image size supported by the first sub-model; it can be arbitrary. For example, the first size can be 224×224, thus meeting the processing requirements of the lightweight first sub-model.

[0071] Step 3022: The first fault image is processed by the third fault detection model to obtain the second probability that the first fault image belongs to each preset type.

[0072] In some embodiments, the size of the first fault image is adjusted by the preprocessing layer of the third fault detection model to obtain a first fault image with a second size; wherein the first size and the second size are different; the second probability of the first fault image with the second size belonging to each of the preset types is determined by the second sub-model of the third fault detection model; wherein the first sub-model and the second sub-model are different.

[0073] Here, the preprocessing layer (i.e., the resize layer) of the third fault detection model is responsible for converting the first fault image into an image with a second size. In some embodiments, the second size is larger than the first size; for example, the second size is 448×448 and the first size is 224×224.

[0074] Understandably, the recognition results of the same image at different sizes, when input into the same sub-model, may differ because the size affects the extraction of image features. Therefore, in this embodiment, a first fault image of a first size is input into the first sub-model, and a first fault image of a second size is input into the second sub-model. Thus, the recognition results output by each sub-model are fused through a fusion layer. Based on this result, a more accurate target type can be obtained through the output layer, further improving the detection accuracy of the second fault detection model, thereby improving the accuracy of the target type. This, on the one hand, reduces the number of times the first fault image needs to be manually labeled, saving labor costs; on the other hand, it improves the detection accuracy of the finally trained second target model, thereby improving the detection accuracy of the first fault detection model.

[0075] In some embodiments, the depth of the second sub-model is greater than the depth of the first sub-model. The second sub-model may include one or more neural network structures; similarly, there is no limitation on the number of neural network structures included in the first sub-model.

[0076] For example, such as Figure 5 As shown, the second fault detection model 50 includes: a first sub-model 501, a second sub-model 502, a fusion layer 503, and an output layer 504. Both the first sub-model 501 and the second sub-model 502 include a convolutional neural network. The first sub-model 501 includes 3 convolutional / activation layers (Conv / ReLU), 3 pooling layers (MaxPooling 3×3), and 1 fully connected / SoftMax layer. The second sub-model 502 includes 4 convolutional layers, 4 pooling layers, and 1 fully connected / SoftMax layer. In the diagram, k, n, and s represent the parameters of the convolutional layers: k represents the kernel size, n represents the number of channels, and s represents the stride. For example, "k11n48s4" in the diagram indicates a kernel size of 11, 48 channels, and a stride of 4. The first sub-model 501 supports image input of 224×224×3, where 3 refers to the number of color channels, and the second sub-model 502 supports image input of 448×448×3.

[0077] Step 3023: Through the fusion layer, the first probability and the second probability corresponding to the preset type are fused to obtain the third probability that the first fault image belongs to the preset type.

[0078] In this application, the method of fusion is not limited. The third probability can be the average of the first and second probabilities, or it can be a weighted average of the first and second probabilities. For example, if the first probability and the second probability that the first faulty image belongs to the mosaic image are 50% and 68% respectively, the weight of the first probability is 0.4, and the weight of the second probability is 0.6, then the fused third probability is 54.4%.

[0079] Step 3024: Through the output layer, determine the third probability that meets the conditions from the third probabilities corresponding to each of the preset types, and output the preset type corresponding to the third probability that meets the conditions as the target type of the first fault image.

[0080] In this embodiment, the target type is obtained by fusing the identification results of various fault detection models, thus improving the accuracy of the target type and reducing the cost of manual annotation. In this application, the condition can be the largest of the third probabilities.

[0081] In step 303, the second electronic device labels the first fault image based on the target type of the first fault image.

[0082] Understandably, the detection accuracy of the second fault detection model is higher than that of the first fault detection model. Therefore, in order to avoid false detections by the first electronic device, the fault type (i.e., target type) of the first fault image needs to be determined again by the second electronic device based on the second fault detection model. This can reduce the number of cases where the first fault image is labeled incorrectly, thereby making the detection accuracy of the second fault detection model trained in the end better, and thus improving the detection accuracy of the first fault detection model on the first electronic device.

[0083] In some embodiments, the second electronic device can directly label the target type as a label for the first fault image, thus saving the cost of manual labeling.

[0084] In other embodiments, the second electronic device can also label the first fault image with manual assistance. Specifically, such as Figure 6 As shown, the second electronic device can implement step 303 through the following steps 3031 to 3033:

[0085] Step 3031: Present the first fault image and its target type.

[0086] Understandably, the second electronic device can display the first fault image and the target type of that image on the interface. This allows the annotator to determine if the target type is correct; if not, the annotator can change the target type.

[0087] Step 3032: Receive a change instruction, the change instruction being used to instruct the target type of the first fault image to be updated to a specified type;

[0088] Step 3033: Based on the change instruction, label the first fault image as the specified type.

[0089] In this embodiment, instead of directly labeling the target type as the label of the first fault image, the incorrect target type is corrected manually, and then the corresponding image is labeled based on the specified type. This can further improve the accuracy of the label, thereby improving the detection accuracy of the second fault detection model, and thus improving the detection accuracy of the monitoring end.

[0090] In step 304, the second electronic device trains the second fault detection model based on at least one first fault image and the labels of each of the first fault images to obtain a second target model; wherein the second target model includes the first target model.

[0091] In some embodiments, the second electronic device may train the second fault detection model directly based on these first fault images and their corresponding labels; in other embodiments, the second electronic device may also merge these first fault images and their corresponding labels with an existing sample dataset to obtain a new sample dataset; and then train the second fault detection model based on this new sample dataset. Specifically, as... Figure 7 As shown, the second electronic device can implement step 304 through steps 3041 to 3045 of the following embodiment:

[0092] Step 3041: Divide the at least one first fault image, the labels of each first fault image, and the at least one stored second fault image and the labels of each second fault image into a training set and a validation set; wherein the training set includes multiple training subsets.

[0093] In some embodiments, the training set comprises a larger percentage than the validation set. For example, the training set may comprise 80% of the total set, and the validation set may comprise 20%.

[0094] It should be noted that the stored second fault image can be a fault image previously uploaded by the first electronic device but not yet used in training, or an image that has been used in training.

[0095] Step 3042: Train the current second fault detection model using the training subset to obtain the trained second fault detection model;

[0096] Step 3043: Verify the detection accuracy of the trained second fault detection model using the validation set.

[0097] In other words, the fault images in the validation set are input into the second fault detection model to obtain the corresponding fault types, and then the fault type is verified to be consistent with the corresponding label. If they are consistent, the fault type is correct; if they are inconsistent, the fault type is incorrect. In this way, the validation results of each fault image in the validation set are obtained. Based on these validation results, the detection accuracy of the second fault detection model can be obtained.

[0098] Step 3044: Determine whether the detection accuracy is less than a specific threshold; if the detection accuracy is greater than or equal to the specific threshold, proceed to step 3045; if the detection accuracy is less than the specific threshold, return to step 3042 and retrain the trained second fault detection model using another training subset until the detection accuracy of the trained second fault detection model is greater than or equal to the specific threshold.

[0099] In this application, the value of the specific threshold is not limited and can be configured according to the actual application. For example, the specific threshold is 98%, but it can also be other values.

[0100] Step 3045: Use the trained second fault detection model as the second target model.

[0101] Understandably, the second objective model includes the first objective model, that is, the model obtained by training the first fault detection model.

[0102] Related solutions for video surveillance fault detection include an automatic fault diagnosis and proactive inspection system for video surveillance under a dedicated network and a video quality diagnostic method based on deep learning. The automatic fault location and inspection system for video surveillance networks built on dedicated networks, based on the configuration of optical line terminals and multiple capability interfaces provided by the network management platform, can accurately locate faulty nodes in the dedicated network. Simultaneously, the system uses a Geographic Information System (GIS) to mark the coordinates of monitoring points on a GIS map, allowing maintenance personnel to quickly process faults and conduct inspections using the system's mobile applications. For different video surveillance platforms, adapter modules provide different interfaces to connect the fault location information of the monitoring point network to the video surveillance platform, integrate it with the video stream, and display it on the display module, allowing the final monitoring personnel to clearly understand the cause of the fault.

[0103] The deep learning-based video quality diagnosis method analyzes the faults in surveillance video images, establishes the correspondence between front-end acquisition faults and video image quality, collects and creates a comprehensive dataset of surveillance video faults, and uses deep learning models to extract more advanced and comprehensive features of abnormal videos, thereby training a more accurate fault detection model with stronger universality in practical applications.

[0104] However, fault detection algorithms for surveillance videos are generally cloud-based or edge-based. While edge-based fault detection models can achieve real-time detection of monitoring faults, model iteration is more difficult. Cloud-based models, on the other hand, cannot detect camera fault status in real time due to network latency and other factors, nor can they obtain edge-based fault data to update the model.

[0105] Based on this, the following will describe an exemplary application of the embodiments of this application in a practical application scenario.

[0106] This application provides an intelligent fault detection method for surveillance videos based on cloud collaboration. An online or offline surveillance device with fault detection capabilities is used as an edge-side terminal device (i.e., a first electronic device). Fault images detected by the surveillance device are saved as images, transmitted to a web server, and then imported into a cloud platform. On the cloud platform, a multi-resolution convolutional neural network (i.e., an example of a second fault detection model) is used to perform secondary verification, cleaning, and annotation of the received fault images. The annotated data can assist in the training and optimization of the multi-resolution convolutional neural network. Some parameters of the multi-resolution convolutional neural network deployed on the cloud platform can be shared with the miniaturized model deployed on the edge-side terminal device (i.e., the first fault detection model). Branches of the multi-resolution convolutional neural network can be directly distributed to the edge-side terminal device via the network, thereby enabling iteration and updating of the edge-side model.

[0107] like Figure 8As shown, this video surveillance fault detection method utilizes an edge computing architecture of "fault-detecting monitoring equipment + cloud platform." The monitoring equipment includes surveillance cameras and fault detection devices. The fault detection devices detect fault images in the monitoring video using a first fault detection model and update their own first fault detection model parameters based on the received model parameters of a first target model. An independently built upper-layer intelligent cloud platform continuously collects video or image data from the fault-detecting monitoring equipment and periodically aggregates the acquired data to the cloud platform. The cloud platform performs secondary verification and correction of the data and trains and optimizes a multi-resolution convolutional neural network model based on the dataset. Partial parameter sharing exists between the cloud platform and the edge-side neural network model. The trained model algorithm on the cloud platform is branched and directly transmitted to the monitoring equipment via 5G technology, completing a self-learning closed loop.

[0108] In implementation, online or offline monitoring devices are treated as edge-side terminal devices. The edge-side terminal devices are equipped with miniaturized fault detection convolutional neural networks (CNNs), i.e., the first fault detection model, for real-time processing. The edge-side CNN model supports image input resolutions of 224×224. A cloud platform and related servers (i.e., an example of a second electronic device) are built and the relevant operating environment is configured. A large-scale algorithm model with higher complexity (i.e., the second fault detection model) is deployed on the cloud platform, such as a multi-resolution convolutional neural network (MRCNN), which supports image input resolutions of 224×224 and 448×448.

[0109] It should be noted that, given that low-resolution and high-resolution images respectively carry local detail features and large-scale object features, MRCNN here extends CNN by feeding the two different input images of different sizes into two different CNNs. The deeper CNN is responsible for extracting global features and large-scale objects, while the shallower CNN is responsible for extracting local details or small-scale objects. Compared to single-branch CNNs, MRCNN achieves higher classification accuracy.

[0110] Specifically, this structure consists of two independent CNNs with different depths. One CNN includes three convolutional layers, and the other includes four. The deeper CNN receives an input image of 448×448×3, while the shallower CNN receives an input image of 224×224×3. After processing by the two CNNs, the output feature maps are 6×6×256 and 6×6×128, respectively. The probabilities of the corresponding categories are obtained through fully connected layers and softmax layers. The final classification result is obtained by taking the arithmetic mean of the classification results from the two networks. In practice, the structure of the two branch models is not limited to the structure used in this embodiment and can be replaced according to actual needs.

[0111] To save on the training process and enhance cloud-edge collaboration, some parameters of the multi-resolution convolutional neural network deployed on the cloud platform can be shared with the miniaturized model deployed on the edge terminal device. That is, the miniaturized model deployed on the edge terminal device is a single branch of the MRCNN deployed on the cloud platform.

[0112] For example, Figure 5 As shown, the multi-resolution convolutional neural network of the cloud platform includes sub-model 501 and sub-model 502, wherein the miniaturized sub-model 501 can also be used as a miniaturized model on the edge side.

[0113] Step 901: Resize the fault images of the monitoring video detected by the edge-side miniaturized model and store them as fault images with a resolution of 448×448 and 224×224 (in .png or .jpg format), and transmit the saved fault images to the cloud platform.

[0114] Step 902: Utilize the cloud platform to perform secondary verification, cleaning, and annotation of the faulty images, obtaining new data. Merge the new data with the existing dataset to obtain a new dataset, which is then divided into two parts: a training set and a validation set, accounting for 80% and 20% respectively. The dataset contains 10 types of data labels: normal images, blurred images, overly bright images, overly dark images, color-distorted images, mosaic images, noisy images, images with camera occlusion, striped images, and images with abnormal camera angles.

[0115] Step 903: Based on the training set generated in step 902, the model is trained on the cloud platform to obtain the trained deep learning algorithm, namely the trained multi-resolution convolutional neural network. During training, hyperparameters such as batch training size, learning rate, and learning rate decay rate need to be adjusted. The fault detection accuracy is continuously improved by adjusting the model parameters.

[0116] Step 904: Input the validation set generated in step 902 into the multi-resolution convolutional neural network trained in step 903, and compare the validation results with the data labels. If the validation set accuracy is lower than 98%, repeat the training step in step 903; if the validation set accuracy is higher than 98%, the model accuracy is considered to have met the standard, and the training ends.

[0117] Step 905: Since some parameters of the multi-resolution convolutional neural network deployed on the cloud platform can be shared with the miniaturized model deployed on the edge terminal device, the shallower CNN branch in the best MRCNN model trained on the cloud platform can be directly distributed to the edge terminal device through the network.

[0118] In this embodiment, 1. An edge computing architecture consisting of a monitoring device with fault detection capabilities and a cloud platform is constructed; 2. The monitoring device with fault detection capabilities can quickly process fault data and periodically upload the fault data to the cloud platform. The multi-resolution convolutional neural network deployed on the cloud platform can perform secondary confirmation and correction of the detection results obtained by the monitoring device on the edge side; 3. Some parameters of the multi-resolution convolutional neural network deployed on the cloud platform can be shared with the miniaturized model deployed on the terminal device on the edge side; 4. The shallower branches in the multi-resolution convolutional neural network trained on the cloud platform can be directly distributed to the terminal device on the edge side through the network to achieve a closed loop of cloud-edge collaboration and realize efficient and rapid deployment of the model.

[0119] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0120] Based on the foregoing embodiments, this application provides a model update device, which includes the included modules and the units included in each module, which can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0121] Figure 9 This is a schematic diagram of the model update device according to an embodiment of this application, as shown below. Figure 9 As shown, the model update device 90 includes:

[0122] The first receiving module 901 is used to receive a first fault image sent by the first electronic device; wherein the first fault image is detected by the first electronic device from the monitoring video through a first fault detection model;

[0123] Training module 902 is used to train at least one stored first fault detection model based on at least one of the first fault images to obtain a first target model;

[0124] The first sending module 903 is used to send the model parameters of the first target model to the first electronic device so that the first electronic device can update the first fault detection model.

[0125] In some embodiments, the training module 902 is configured to: perform fault detection on the first fault image using a stored second fault detection model to obtain the target type of the first fault image; wherein the first fault detection model is a branch of the second fault detection model; label the first fault image based on the target type of the first fault image; and train the second fault detection model based on at least one first fault image and the labels of each of the first fault images to obtain a second target model; wherein the second target model includes the first target model.

[0126] In some embodiments, the second fault detection model includes a third fault detection model, a first fault detection model, a fusion layer, and an output layer, wherein the third fault detection model has a different structure from the first fault detection model; the training module 902 is configured to: process the first fault image using the first fault detection model to obtain a first probability that the first fault image belongs to each preset type; process the first fault image using the third fault detection model to obtain a second probability that the first fault image belongs to each preset type; fuse the first and second probabilities corresponding to the preset types using the fusion layer to obtain a third probability that the first fault image belongs to the preset type; and determine a third probability that satisfies a condition from the third probabilities corresponding to each preset type using the output layer, and output the preset type corresponding to the third probability that satisfies the condition as the target type of the first fault image.

[0127] In some embodiments, the training module 902 is configured to: adjust the size of the first fault image through the preprocessing layer of the first fault detection model to obtain a first fault image with a first size; and determine the first probability that the first fault image with the first size belongs to each of the preset types through the first sub-model of the first fault detection model.

[0128] In some embodiments, the training module 902 is configured to: adjust the size of the first fault image through the preprocessing layer of the third fault detection model to obtain a first fault image with a second size; wherein the first size and the second size are different; and determine the second probability that the first fault image with the second size belongs to each of the preset types through the second sub-model of the third fault detection model; wherein the first sub-model and the second sub-model are different.

[0129] In some embodiments, the training module 902 is configured to: present the first fault image and its target type; receive a change instruction, the change instruction indicating that the target type of the first fault image be updated to a specified type; and, based on the change instruction, label the first fault image with the specified type.

[0130] In some embodiments, the training module 902 is configured to: divide the at least one first fault image, the labels of each of the first fault images, and the at least one stored second fault image and the labels of each of the second fault images into a training set and a validation set; wherein the training set includes multiple training subsets; train the current second fault detection model using the training subsets to obtain a trained second fault detection model; verify the detection accuracy of the trained second fault detection model using the validation set; if the detection accuracy is less than a specific threshold, retrain the trained second fault detection model using another training subset until the detection accuracy of the trained second fault detection model is greater than or equal to the specific threshold, and use the second fault detection model with a detection accuracy greater than or equal to the specific threshold as the second target model.

[0131] In some embodiments, the first electronic device includes a monitoring device, a model update device 90 is applied to a second electronic device, the second electronic device is configured to run a cloud platform, the cloud platform being used to train a stored first fault detection model based on at least one of the first fault images.

[0132] This application embodiment further provides a model update device. Figure 10 This is a schematic diagram of the model update device according to an embodiment of this application, as shown below. Figure 10 As shown, the model update device 100 includes:

[0133] Fault detection module 1001 is used to detect fault images in the monitoring video through a first fault detection model to obtain a first fault image;

[0134] The second sending module 1002 is used to send the first fault image to the second electronic device; wherein the first fault image is used by the second electronic device to train the stored first fault detection model;

[0135] The second receiving module 1003 is used to receive model parameters of the first target model sent by the second electronic device; the first target model is obtained by the second electronic device training the first fault detection model based on at least one first fault image;

[0136] The parameter update module 1004 is used to update the first fault detection model based on the model parameters of the first target model.

[0137] In some embodiments, the model update device 100 is applied to a first electronic device, the first electronic device including a monitoring device, and the second electronic device is configured to run a cloud platform for training a stored first fault detection model.

[0138] In some embodiments, the fault detection module 1001 is configured to: adjust the size of the video frames of the monitoring video through the preprocessing layer of the first fault detection model to obtain video frames with a first size; determine, through the first sub-model of the first fault detection model, the fourth probability of the video frames with the first size belonging to various preset types; and determine, based on the fourth probability of each preset type, whether the video frame is a fault image.

[0139] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0140] It should be noted that the module division of the model update device described in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0141] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0142] This application provides an electronic device. Figure 11 This is a schematic diagram of the hardware entity of the electronic device according to an embodiment of this application, such as... Figure 11 As shown, the electronic device 110 includes a memory 1101 and a processor 1102. The memory 1101 stores a computer program that can run on the processor 1102. When the processor 1102 executes the program, it implements the steps in the method provided in the above embodiments.

[0143] It should be noted that the memory 1101 is configured to store instructions and applications executable by the processor 1102, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 1102 and the various modules in the electronic device 110. It can be implemented by flash memory or random access memory (RAM).

[0144] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0145] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0146] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0147] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0148] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0149] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0151] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0153] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0154] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0155] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0156] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0157] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0158] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model update method, characterized in that, The method includes: Receive a first fault image sent by a first electronic device; wherein the first fault image is detected by the first electronic device from a monitoring video through a first fault detection model; Based on at least one of the first fault images, at least the stored first fault detection model is trained to obtain a first target model; The model parameters of the first target model are sent to the first electronic device so that the first electronic device can update the first fault detection model; The step of training a first target model based on at least one of the first fault images, and training at least one stored first fault detection model, includes: The first fault image is detected by using the stored second fault detection model to obtain the target type of the first fault image; wherein, the first fault detection model is a branch of the second fault detection model; Label the first fault image based on the target type of the first fault image; The second fault detection model is trained based on at least one of the first fault images and the labels of each of the first fault images to obtain a second target model; wherein the second target model includes the first target model.

2. The method according to claim 1, characterized in that, The second fault detection model includes a third fault detection model, a first fault detection model, a fusion layer, and an output layer. The third fault detection model has a different structure from the first fault detection model. The step of performing fault detection on the first fault image using the stored second fault detection model to obtain the target type of the first fault image includes: The first fault image is processed by the first fault detection model to obtain the first probability that the first fault image belongs to each preset type. The first fault image is processed using the third fault detection model to obtain the second probability that the first fault image belongs to each preset type. The fusion layer fuses the first probability and the second probability corresponding to the preset type to obtain the third probability that the first fault image belongs to the preset type. The output layer determines the third probability that meets the conditions from the third probabilities corresponding to each preset type, and outputs the preset type corresponding to the third probability that meets the conditions as the target type of the first fault image.

3. The method according to claim 2, characterized in that, The step of processing the first fault image using the first fault detection model to obtain the first probability that the first fault image belongs to each preset type includes: The size of the first fault image is adjusted by the preprocessing layer of the first fault detection model to obtain a first fault image with a first size; The first probability of the first fault image with a first size belonging to each of the preset types is determined by the first sub-model of the first fault detection model. The process of processing the first fault image using the third fault detection model to obtain the second probability that the first fault image belongs to each preset type includes: The size of the first fault image is adjusted by the preprocessing layer of the third fault detection model to obtain a first fault image with a second size; wherein the first size and the second size are different. The second sub-model of the third fault detection model determines the second probability that the first fault image with the second size belongs to each of the preset types; wherein the first sub-model and the second sub-model are different.

4. The method according to claim 1, characterized in that, The step of labeling the first fault image based on its target type includes: Present the first fault image and its target type; Receive a change instruction, the change instruction being used to instruct the target type of the first fault image to be updated to a specified type; Based on the change instruction, the label of the first fault image is marked as the specified type.

5. The method according to claim 1, characterized in that, The step of training the second fault detection model based on at least one first fault image and the labels of each first fault image to obtain a second target model includes: The at least one first fault image, the labels of each first fault image, and the at least one stored second fault image and the labels of each second fault image are divided into a training set and a validation set; wherein the training set includes multiple training subsets. The current second fault detection model is trained using the training subset to obtain the trained second fault detection model; The detection accuracy of the trained second fault detection model is verified using the validation set. If the detection accuracy is less than the first threshold, the trained second fault detection model is retrained using another training subset until the detection accuracy of the trained second fault detection model is greater than or equal to the first threshold. The second fault detection model with a detection accuracy greater than or equal to the first threshold is then used as the second target model.

6. The method according to any one of claims 1 to 5, characterized in that, The first electronic device includes a monitoring device, the method is applied to a second electronic device, the second electronic device is configured to run a cloud platform, the cloud platform is used to train a stored first fault detection model based on at least one of the first fault images.

7. A model update method, characterized in that, The method includes: The first fault image is obtained by detecting fault images in the monitoring video using the first fault detection model. The first fault image is sent to the second electronic device; wherein the first fault image is used by the second electronic device to train the stored first fault detection model; The system receives model parameters of a first target model sent by the second electronic device. The second target model includes the first target model. The second target model is obtained by the second electronic device training a second fault detection model based on at least one first fault image and the labels of each first fault image. The labels of the first fault images are marked by the second electronic device based on the target type of the first fault images. The target type of the first fault images is obtained by the second electronic device performing fault detection on the first fault images using the stored second fault detection model. The first fault detection model is a branch of the second fault detection model. The first fault detection model is updated based on the model parameters of the first target model.

8. The method according to claim 7, characterized in that, The method is applied to a first electronic device, the first electronic device including a monitoring device, and a second electronic device configured to run a cloud platform, the cloud platform being used to train a stored first fault detection model.

9. A model update device, characterized in that, The device includes: A first receiving module is configured to receive a first fault image sent by a first electronic device; wherein the first fault image is detected by the first electronic device from a monitoring video through a first fault detection model; The training module is used to train at least one stored first fault detection model based on at least one of the first fault images to obtain a first target model. The first sending module is used to send the model parameters of the first target model to the first electronic device so that the first electronic device can update the first fault detection model. The training module trains at least one of the first fault images and at least one stored first fault detection model to obtain a first target model, including: The first fault image is detected by using the stored second fault detection model to obtain the target type of the first fault image; wherein, the first fault detection model is a branch of the second fault detection model; Label the first fault image based on the target type of the first fault image; The second fault detection model is trained based on at least one of the first fault images and the labels of each of the first fault images to obtain a second target model; wherein the second target model includes the first target model.

10. A model update device, characterized in that, The device includes: The fault detection module is used to detect fault images in the monitoring video through the first fault detection model and obtain the first fault image; The second sending module is used to send the first fault image to the second electronic device; wherein the first fault image is used by the second electronic device to train the stored first fault detection model; The second receiving module is used to receive model parameters of the first target model sent by the second electronic device; the second target model includes the first target model, and the second target model is obtained by the second electronic device training a second fault detection model based on at least one first fault image and the labels of each first fault image, the labels of the first fault images are labeled by the second electronic device based on the target type of the first fault images, and the target type of the first fault images is obtained by the second electronic device through the stored second fault detection model to perform fault detection on the first fault images, wherein the first fault detection model is a branch of the second fault detection model; The parameter update module is used to update the first fault detection model based on the model parameters of the first target model.

11. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 6, or when the processor executes the program, it implements the method according to claim 7 or 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6, or when the computer program is executed by a processor, it implements the method as described in claim 7 or 8.

Images