Target detection method and device, electronic device, and storage medium

Abstract

The application provides a target detection method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: an edge node acquires a plurality of sample images, then performs inference on each sample image by using a deployed model, obtains a difficult example sample, and sends the difficult example sample to a cloud end; the cloud end performs incremental training on an initial model according to the difficult example sample, obtains a sample model, performs evaluation, and sends a sample model meeting a preset condition to the edge node; the edge node updates the deployed model according to the sample model, and uses the updated deployed model as a trained target detection model; and the edge node acquires a to-be-detected image, inputs the to-be-detected image into the target detection model, and obtains a target detection result. The application automatically trains the target detection model through cloud-edge cooperation, fully utilizes the computing and storage capacity of the cloud end, can timely train and learn according to the generated sample images, and further improves the precision of target detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target detection method and apparatus, electronic device and storage medium. Background Technology

[0002] Object detection is an important component of image detection. The task of object detection is to find all targets (objects) of interest in an image and determine their location and size. Because the same object may have different appearances and shapes, and due to factors such as aging imaging equipment, climate change, and occlusion, the target data in the images generated by the imaging equipment is constantly changing.

[0003] In related technologies, object detection methods improve object detection performance by manually collecting newly generated images in the current environment for training. However, this method wastes a lot of human resources and is difficult to train in a timely manner, affecting the accuracy of object detection. Therefore, how to improve the accuracy of object detection has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to provide a target detection method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of target detection.

[0005] To achieve the above objectives, a first aspect of this application proposes a target detection method applied to a cloud-edge collaborative system, wherein the cloud-edge collaborative system includes edge nodes and a cloud, and the method includes:

[0006] Multiple sample images are obtained from the edge nodes;

[0007] The edge nodes perform inference on each of the sample images using a pre-set deployment model to obtain difficult sample examples, and then send the difficult sample examples to the cloud.

[0008] The cloud platform incrementally trains the pre-set initial model based on the difficult example samples to obtain the sample model.

[0009] The cloud evaluates the sample model and sends the sample model whose evaluation results meet preset conditions to the edge node;

[0010] The edge nodes update the deployment model based on the sample model, and the updated deployment model is used as the trained target detection model.

[0011] The image to be detected is obtained from the edge nodes, and the image to be detected is input into the target detection model to obtain the target detection result.

[0012] In some embodiments, the step of obtaining hard example samples by inferring from each of the sample images using a pre-set deployment model by the edge nodes includes:

[0013] For each sample image, the edge nodes infer the sample image using a pre-set deployment model to obtain at least one predicted bounding box and the confidence level corresponding to the predicted bounding box;

[0014] The difficulty coefficient of the sample image is calculated based on the predicted bounding box and the confidence level corresponding to the predicted bounding box.

[0015] When the difficulty coefficient is greater than the preset difficulty threshold, the sample image is determined to be a difficult sample.

[0016] In some embodiments, after sending the difficult example sample to the cloud, the method further includes:

[0017] The cloud platform performs statistical analysis on the received difficult sample samples to obtain the number of difficult sample samples.

[0018] When the number of difficult examples is greater than or equal to a preset threshold, the process is triggered to obtain the pre-set initial model for incremental training.

[0019] In some embodiments, the step of incrementally training a pre-set initial model using the difficult example samples to obtain a sample model by the cloud includes:

[0020] Obtain the labeled data corresponding to each of the aforementioned difficult example samples;

[0021] A set of difficult examples is determined based on multiple difficult example samples and the corresponding labeled data of the difficult example samples;

[0022] A training sample set is extracted from the set of difficult examples according to a preset training ratio;

[0023] Each training sample in the training sample set is input into the initial model to obtain the first sample detection result corresponding to each training sample.

[0024] Based on the labeled data corresponding to each training sample and the detection result of the first sample, the parameters of the initial model are optimized to obtain the sample model.

[0025] In some embodiments, the step of evaluating the sample model by the cloud and sending the sample model whose evaluation result meets preset conditions to the edge node includes:

[0026] The difficult example set is obtained by removing the difficult example samples that exist in the training sample set;

[0027] Each evaluation sample in the evaluation sample set is input into the sample model to obtain the second sample detection result corresponding to each evaluation sample;

[0028] The evaluation accuracy of the sample model is determined based on the labeled data corresponding to each evaluation sample and the detection results of the second sample.

[0029] When the evaluation accuracy is greater than or equal to a preset accuracy threshold, the sample model is sent to the edge node.

[0030] In some embodiments, before acquiring multiple sample images from the edge nodes, the method further includes:

[0031] The target detection model is obtained from the cloud.

[0032] The cloud platform converts the target detection model into a first format and a second format, respectively, and uses the first format target detection model as the initial model and the second format target detection model as the deployment model.

[0033] The cloud takes the target detection model in the first format as the initial model and sets the training parameters of the initial model, wherein the training parameters include at least one of the following: quantity threshold, training ratio, and accuracy threshold;

[0034] The cloud takes the target detection model in the second format as the deployment model and sends the deployment model to the edge node.

[0035] In some embodiments, after the cloud performs incremental training on a pre-set initial model based on the difficult example samples to obtain a sample model, the method further includes:

[0036] The cloud platform converts the sample model from a first format to a second format and evaluates the sample model in the second format, so as to send the sample model that meets the preset conditions to the edge node.

[0037] To achieve the above objectives, a second aspect of this application provides a target detection device applied to the cloud-edge collaborative system, the cloud-edge collaborative system including edge nodes and a cloud, the target detection device being connected to both the edge nodes and the cloud, the device comprising:

[0038] The image acquisition module is used to control the edge nodes to acquire multiple sample images;

[0039] The edge inference module is used to control the edge nodes to infer each of the sample images through a pre-set deployment model, obtain difficult sample examples, and send the difficult sample examples to the cloud.

[0040] The incremental training module is used to control the cloud to perform incremental training on the pre-set initial model based on the difficult example samples to obtain the sample model;

[0041] The model evaluation module is used to control the cloud to evaluate the sample model and send the sample model whose evaluation results meet the preset conditions to the edge node;

[0042] The model update module is used to control the edge nodes to update the deployment model according to the sample model, and use the updated deployment model as the trained target detection model;

[0043] The target detection module is used to control the edge nodes to acquire the image to be detected and input the image to be detected into the target detection model to obtain the target detection result.

[0044] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0046] The object detection method, apparatus, electronic device, and storage medium proposed in this application are applied to a cloud-edge collaborative system. This method is implemented through edge nodes and the cloud within the cloud-edge collaborative system. First, the edge nodes acquire multiple sample images and infer from these images using a pre-set deployment model to obtain difficult examples. These difficult examples are then sent to the cloud. Next, the cloud incrementally trains a pre-set initial model based on the difficult examples to obtain sample models. These sample models are then evaluated, and those meeting preset evaluation conditions are sent to the edge nodes. The edge nodes then update the deployment model based on the sample models, resulting in the trained object detection model. Finally, the edge nodes acquire the image to be detected and input it into the object detection model to obtain the object detection result. This application's embodiment automatically trains, evaluates, and updates the object detection model through cloud-edge collaboration, and applies the trained model at the edge nodes. It fully utilizes the computing and storage capabilities of the cloud, enabling timely training based on newly generated sample images, shortening the update cycle of the object detection model, and further improving the accuracy of object detection. Attached Figure Description

[0047] Figure 1 This is a flowchart of the main target detection method provided in the embodiments of this application;

[0048] Figure 2 This is a flowchart of the edge inference process of the target detection method provided in the embodiments of this application;

[0049] Figure 3 This is a flowchart illustrating the triggering of incremental training for the target detection method provided in this application embodiment;

[0050] Figure 4 This is a flowchart of the incremental training of the target detection method provided in the embodiments of this application;

[0051] Figure 5 This is a flowchart of the model evaluation of the target detection method provided in the embodiments of this application;

[0052] Figure 6 This is a flowchart of the model deployment of the target detection method provided in the embodiments of this application;

[0053] Figure 7 This is a flowchart illustrating the format conversion of the target detection method provided in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram of the cloud-edge collaborative system provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the target detection device provided in the embodiments of this application;

[0056] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] First, let's analyze some of the terms used in this application:

[0061] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and applications to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert devices. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0062] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0063] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0064] Cloud-edge collaboration refers to the coordinated operation of edge computing and cloud computing to jointly unlock the value of data. Traditional cloud-edge collaboration primarily involves terminal devices generating data or task requests, which are then uploaded to edge servers via the edge network. The edge servers, located in the edge computing center, then execute the computational tasks. For computationally intensive and complex tasks, the edge computing center migrates upwards through the core network to the cloud computing center. After the cloud computing center completes the big data analysis, the results and data are stored there, or the computation results, optimized business rules, and models are distributed to the edge computing center via the core network. The edge computing center then transmits the computation results downwards through the edge network to the terminal devices. The edge computing then performs business execution and optimization based on the new business rules issued by the cloud computing center, thus achieving cloud-edge collaboration.

[0065] Object detection is a crucial component of image detection. Its main task is to identify all objects of interest in an image and determine their location and size. Due to the diverse appearances, shapes, and poses of various objects, coupled with interference from factors such as lighting and occlusion during imaging, object detection remains one of the most challenging problems in machine vision. Object detection is widely used in robot navigation, intelligent video surveillance, industrial inspection, aerospace, and many other fields. Furthermore, object detection is a prerequisite algorithm for many vision tasks, playing a vital role in subsequent tasks such as face recognition, gait recognition, crowd counting, and instance segmentation.

[0066] In related technologies, object detection methods improve object detection performance by manually collecting newly generated images in the current environment for training. However, this method wastes a lot of human resources and is difficult to train in a timely manner, affecting the accuracy of object detection. Therefore, how to improve the accuracy of object detection has become an urgent technical problem to be solved.

[0067] Based on this, embodiments of this application provide a target detection method and apparatus, electronic device and storage medium, aiming to improve the accuracy of target detection.

[0068] The target detection method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the target detection method in this application is described.

[0069] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application devices that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0070] Foundational artificial intelligence technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive devices, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer device environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, microprocessor-based devices, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above devices, etc. This application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0073] Figure 1 This is a flowchart of the main target detection method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S100 to S600.

[0074] Step S100: Obtain multiple sample images from edge nodes.

[0075] Step S200: The edge nodes perform inference on each sample image through a pre-set deployment model to obtain difficult sample images, and then send the difficult sample images to the cloud.

[0076] Step S300: The cloud performs incremental training on the pre-set initial model based on the difficult example samples to obtain the sample model.

[0077] Step S400: The cloud evaluates the sample model and sends the sample model whose evaluation results meet the preset conditions to the edge node.

[0078] Step S500: The edge nodes update the deployment model based on the sample model, and use the updated deployment model as the trained target detection model.

[0079] Step S600: Obtain the image to be detected from the edge nodes and input the image to be detected into the target detection model to obtain the target detection result.

[0080] It should be noted that the target detection method provided in this application embodiment is applied to a cloud-edge collaborative system, which includes edge nodes and the cloud.

[0081] Steps S100 to S600 as illustrated in this embodiment involve: First, edge nodes acquire multiple sample images and infer from these images using a pre-set deployment model to obtain difficult sample images. These difficult sample images are then sent to the cloud. Next, the cloud incrementally trains a pre-set initial model based on the difficult sample images to obtain a sample model. The sample model is then evaluated, and models whose evaluation results meet preset conditions are sent to the edge nodes. The edge nodes then update the deployment model based on the sample model, resulting in the trained target detection model. Finally, the edge nodes acquire the image to be detected and input it into the target detection model to obtain the target detection result. This embodiment automatically trains, evaluates, and updates the target detection model through cloud-edge collaboration and applies the trained model at the edge nodes. This fully utilizes the computing and storage capabilities of the cloud, enabling timely training based on newly generated sample images. This process does not require manual intervention, shortens the update cycle of the target detection model, and further improves the accuracy of target detection.

[0082] In step S100 of some embodiments, the edge node can acquire sample images from devices such as cameras or video cameras, or it can acquire sample images by cropping video. By acquiring multiple sample images, rich sample materials can be provided for the training of the subsequent object detection model.

[0083] In step S200 of some embodiments, the deployment model is an untrained target detection model. Edge computing is close to end users and geographically dispersed, which can support application services with low latency, location awareness, high mobility and high QoS. Therefore, the deployment model is deployed on edge nodes and sample images are input into the deployment model. Due to the influence of external environmental factors, some sample images may have difficulty detecting the correct target. These sample images are sent to the cloud as hard examples for incremental training, so that the finally trained target detection model can identify hard examples and improve the detection accuracy of the target detection model.

[0084] In step S300 of some embodiments, cloud computing technology provides powerful computing capabilities with cheap and abundant computing servers, which can provide users and applications with rich computing and storage resources that can be accessed on demand. Therefore, the initial model is deployed in the cloud and incrementally trained by the cloud. Incremental training is also known as incremental learning. Incremental training means that a learning system can continuously learn new knowledge from new samples and can retain most of the previously learned knowledge. Therefore, the model obtained through incremental training can detect difficult samples and can also perform object detection on other existing sample images.

[0085] In step S400 of some embodiments, the cloud evaluates the sample model. When the evaluation result meets the preset requirements, that is, when the evaluation effect of the sample model meets the preset requirements, the sample model is sent to the edge node. The evaluation is to evaluate the performance of the sample model obtained by incremental training. If the evaluation result of the sample model meets the requirements, then the final target detection model can detect difficult samples, and its target detection capability is improved.

[0086] In step S500 of some embodiments, at the edge node, the deployment model is updated according to the sample model, so that the updated deployment model is a sample model that meets the preset evaluation requirements, and the updated deployment model is used as a trained target detection model, so that the deployment model can perform target detection, shortening the model update cycle and further improving the detection accuracy of the target detection model.

[0087] In step S600 of some embodiments, the edge node acquires the image to be detected and inputs the image to be detected into the target detection model to obtain the target detection result. This process is an application of the trained target detection model. After the above training, the edge node performs target detection on the image to be detected, and the obtained target detection result is more accurate.

[0088] It's important to note that edge computing refers to processing, analyzing, and storing data closer to its generation, enabling rapid and near real-time analysis and response. The cloud-native concept aims to maximize the utilization of cloud resources, leveraging the elasticity and reliability of distributed systems to allow systems to fully benefit from cloud computing. Cloud-edge collaboration achieves synergistic interaction between edge computing and cloud computing, jointly unlocking data value. High-computation, complex tasks are migrated from edge nodes to the cloud via the core network. After big data analysis is completed in the cloud, the results and data are stored there, or the computation results, optimized business rules, and models are distributed to edge nodes via the core network. Edge nodes then transmit the computation results down to terminal devices via the edge network. Edge computing executes and optimizes processes based on new business rules issued by the cloud, thus achieving cloud-edge collaboration. This collaboration achieves comprehensive synergy across three domains: resources, management, and application services.

[0089] It should be noted that, referring to Figure 8 , Figure 8 This is a schematic diagram of the cloud-edge collaborative system provided in this application embodiment. The cloud-edge collaborative system includes one or more cloud nodes and multiple edge nodes, which form a cluster. Data is continuously generated at the edge nodes, and the cloud continuously monitors the newly generated data. By configuring some trigger rules, it determines whether to automatically start training, evaluation, and deployment, and continuously improves model performance. The object detection method provided in this application embodiment continuously monitors newly generated sample images in the cloud and determines whether to automatically start training, evaluation, and deployment by configuring training parameters, thereby continuously improving model performance. The deployed model at the edge node is a pre-trained object detection model, which can be configured for users. Users can achieve object detection and further improve object detection accuracy by applying the object detection model.

[0090] Please see Figure 2 In some embodiments, step S200 may include, but is not limited to, steps S210 to S230:

[0091] Step S210: For each sample image, the edge nodes perform inference on the sample image through a pre-set deployment model to obtain at least one prediction box and the confidence level corresponding to the prediction box.

[0092] Step S220: Calculate the difficulty coefficient of the sample image based on the predicted bounding box and the confidence level corresponding to the predicted bounding box.

[0093] Step S230: When the difficulty coefficient is greater than the preset difficulty threshold, the sample image is determined to be a difficult sample.

[0094] In step S210 of some embodiments, the target detection process involves labeling the target to be detected in the sample image using a prediction box. Therefore, the sample image can be input into the deployment model pre-set by the edge nodes for target detection, and at least one prediction box and the confidence level corresponding to the prediction box can be obtained.

[0095] It should be noted that the confidence score is used to characterize the credibility of the predicted bounding box, and the value ranges from 0 to 1. The higher the confidence score, the more likely the target to be detected is to be present in the predicted bounding box.

[0096] In step S220 of some embodiments, the proportion of the number of inference results with confidence levels less than a preset box threshold in a sample image is calculated to obtain the difficulty coefficient of the sample image.

[0097] In step S230 of some embodiments, the difficulty coefficient of the sample image is compared with a preset difficulty threshold. When the difficulty coefficient is greater than the preset difficulty threshold, the sample image is a difficult sample.

[0098] It should be noted that the principle for selecting hard examples in this embodiment is as follows: `box_threshold` (box threshold) and `img_threshold` (hard example threshold) are set. `box_threshold` is used to calculate the hard example coefficient of the sample image, which is the percentage of the number of inference results with confidence scores less than the threshold out of the total output predicted boxes. `img_threshold` is used to determine whether the sample image is a hard example; any image with a hard example coefficient greater than the threshold is considered a hard example sample. The input is a list of predicted boxes, for example, `[bbox1,bbox2,bbox3,....]`, where `bbox` represents a predicted box, and `bbox = [x1,y1,x2,y2,score,label]`, where `x1,y1,x2,y2` represent the coordinates of the predicted box, `score` represents the inference confidence score corresponding to the predicted box, and `label` represents the label corresponding to the predicted box. The value of `score` ranges from 0 to 1. The output result is either `True` or `False`, where `True` represents a hard example sample and `False` represents a non-hard example sample.

[0099] In one embodiment, assuming box_threshold is set to 0.9 and img_threshold is 0.6, after inference on the sample image, there are 10 predicted boxes, that is, the predicted box list has 10 elements, of which 8 elements have a confidence level of less than 0.9. Then the hardness coefficient of the sample image is 8 / 10 = 0.8. Since the hardness coefficient of the sample image is greater than the hardness threshold of 0.6, the sample image is considered a hard sample.

[0100] It should be noted that by deploying the model at the edge nodes to perform object detection on the sample image, at least one predicted bounding box and the confidence score corresponding to the predicted bounding box are obtained. Based on this, the hardness coefficient of the sample image is calculated. Finally, by comparing the hardness coefficient with the preset hardness threshold, it is determined whether the sample image is a hard example. The deployment model directly outputs the sample images that can be distinguished, while using the images that are difficult to distinguish as the sample data for subsequent model training. This enables the trained object detection model to distinguish hard examples, improves the detection accuracy of the model, and thus improves the accuracy of object detection.

[0101] Please see Figure 3 In some embodiments, the interval between step S200 and step S300 may include, but is not limited to, steps S710 to S720:

[0102] Step S710: The cloud performs statistics on the received difficult sample samples to obtain the number of difficult sample samples.

[0103] Step S720: When the number of difficult examples is greater than or equal to a preset threshold, the process of obtaining a pre-set initial model is triggered to perform incremental training on the initial model.

[0104] In step S710 of some embodiments, since model training requires a certain number of samples, the number of difficult sample samples can be obtained by statistically analyzing the received difficult sample samples through the cloud, which is convenient for providing a corresponding number of difficult sample samples for subsequent model training.

[0105] In step S720 of some embodiments, the user can preset a quantity threshold. When the number of difficult sample samples is greater than or equal to the quantity threshold, the user is triggered to obtain the preset initial model for incremental training.

[0106] It should be noted that, in this embodiment of the application, the initial model is incrementally trained by calling TensorFlow functions.

[0107] It should be noted that, in this embodiment of the application, the number of difficult sample cases received is monitored by the cloud. When the number of difficult sample cases is greater than or equal to the number threshold, the initial model is triggered to perform incremental training, which provides sufficient samples for subsequent incremental training and makes the trained object detection model more widely applicable.

[0108] Please see Figure 4 In some embodiments, step S300 may include, but is not limited to, steps S310 to S350:

[0109] Step S310: Obtain the labeled data corresponding to each difficult example sample.

[0110] Step S320: Determine the difficult example set based on multiple difficult example samples and the labeled data corresponding to the difficult example samples.

[0111] Step S330: Extract the training sample set from the hard example set according to the preset training ratio.

[0112] Step S340: Input each training sample in the training sample set into the initial model to obtain the first sample detection result corresponding to each training sample.

[0113] Step S350: Optimize the parameters of the initial model based on the labeled data corresponding to each training sample and the detection result of the first sample to obtain the sample model.

[0114] In step S310 of some embodiments, the annotation data corresponding to the difficult sample is the image obtained after annotating the target to be detected in the difficult sample, or the annotation data is at least one predicted box and the corresponding confidence level obtained after target detection, and the confidence level is 1.

[0115] In step S320 of some embodiments, multiple hard example samples and the labeled data corresponding to the hard example samples are used as a hard example set to facilitate the subsequent training and evaluation of the model.

[0116] In step S330 of some embodiments, the difficult example set includes all difficult example samples received from the cloud. The number of difficult example samples in the difficult example set is multiplied by a preset training ratio, and the number of difficult example samples in the difficult example set corresponding to the multiplication result is selected as the training sample set.

[0117] In step S340 of some embodiments, each training sample in the training sample set is input into the initial model to obtain the first sample detection result corresponding to each training sample, and the parameters of the initial model are optimized in reverse by using the first sample detection result.

[0118] In step S350 of some embodiments, the parameters of the initial model are optimized based on the labeled data corresponding to each training sample and the detection result of the first sample, and the initial model with optimized parameters is used as the sample model.

[0119] It should be noted that a set of difficult examples with a predetermined training ratio is selected as the training sample set in the difficult example set. The initial model is trained in the cloud using the training sample set to optimize the parameters of the initial model and obtain the sample model. This sample model is able to distinguish difficult examples, thereby improving the detection capability of the target detection model.

[0120] Please see Figure 5 In some embodiments, step S400 may also include, but is not limited to, steps S410 to S440:

[0121] Step S410: Remove the difficult examples that are present in the training sample set to obtain the evaluation sample set.

[0122] Step S420: Input each evaluation sample in the evaluation sample set into the sample model to obtain the second sample detection result corresponding to each evaluation sample.

[0123] Step S430: Determine the evaluation accuracy of the sample model based on the labeled data corresponding to each evaluation sample and the detection results of the second sample.

[0124] Step S440: When the evaluation accuracy is greater than or equal to the preset accuracy threshold, the sample model is sent to the edge node.

[0125] In step S410 of some embodiments, the difficult examples in the difficult example set can be used for incremental training and evaluation. The difficult examples corresponding to the training sample set in the difficult example set are removed to obtain the evaluation result, which avoids the duplication of the training sample set and the evaluation sample set and enhances the reliability of the evaluation result.

[0126] In step S420 of some embodiments, each evaluation sample in the evaluation sample set is input into the sample model to obtain the evaluation result, wherein the evaluation result is a hard example sample of object detection including at least one prediction box.

[0127] In step S430 of some embodiments, the labeled data corresponding to the evaluation sample and the detection result of the second sample are compared. The labeled data corresponding to the evaluation sample can cover all the targets to be detected in the evaluation sample. By comparing the two, it can be determined whether the sample model can distinguish the currently input difficult sample. The proportion of the number of evaluation samples that the sample can distinguish to the total number of evaluation samples in the evaluation sample set is used as the evaluation accuracy of the sample model for the evaluation sample set.

[0128] In step S440 of some embodiments, when the evaluation accuracy is greater than or equal to a preset accuracy threshold, the sample model meets the requirements for its detection capability, and therefore, the sample model is sent to the edge node.

[0129] It should be noted that the performance of the sample model is evaluated by assessing the difficult examples in the evaluation sample set. When the accuracy of the sample model on the evaluation sample set is greater than or equal to a preset accuracy threshold, the sample model generated by incremental training meets the requirements for the object detection model, and the cloud controls the sending of the sample model to the edge nodes. The settings of the evaluation process ensure that the object detection model meets the requirements, reducing the possibility of the object detection model having low discrimination ability on difficult examples when directly applied, and making the performance of the object detection model more intuitive.

[0130] In one embodiment, a threshold of 1000 is defined, the training ratio is 0.8, and the evaluation accuracy is 95%. The number of hard example samples received is monitored via the cloud. Once the number of samples is greater than or equal to the set threshold (1000), incremental training of a pre-set initial model is triggered. Assuming the current number of hard example samples is 1000, the number of hard example samples in the training sample set is 800 (sample count * training ratio). An incremental training task is initiated, using 800 hard example samples to incrementally train the initial model, resulting in a sample model. After training, an evaluation task is initiated, using the remaining hard example samples in the hard example set as the evaluation sample set. The evaluation samples from this set are then input into the sample model to evaluate its performance. When the evaluation accuracy of the sample model on the evaluation sample set is greater than or equal to the accuracy threshold, the sample model is sent to the edge node. If the evaluation accuracy is less than 95%, the edge node makes no changes.

[0131] Please see Figure 6 In some embodiments, steps S810 to S840 are included, but are not limited to, preceding step S100:

[0132] Step S810: Obtain the target detection model from the cloud.

[0133] In step S820, the cloud performs format conversion on the target detection model to obtain a first format target detection model and a second format target detection model.

[0134] In step S830, the cloud takes the first format object detection model as the initial model and sets the training parameters of the initial model. The training parameters include at least one of the following: quantity threshold, training ratio, and accuracy threshold.

[0135] In step S840, the cloud sends the second-format target detection model as the deployment model to the edge node.

[0136] In step S810 of some embodiments, the target detection model is an existing target detection model.

[0137] In step S820 of some embodiments, for the target detection model that is incrementally trained using TensorFlow functions, it is necessary to convert the target detection model into a format, resulting in a target detection model in ckpt format and a target detection model in pb format.

[0138] It should be noted that the first format object detection model is in ckpt format, and the second format object detection model is in pb format.

[0139] In step S830 of some embodiments, cloud computing technology provides powerful computing capabilities with cheap and large numbers of computing servers, which can provide users and applications with rich computing and storage resources that can be accessed on demand. Therefore, the first format object detection model, namely the ckpt format object detection model, is deployed in the cloud as the initial model and incrementally trained on the model in the cloud. In addition, the cloud also sets the training parameters of the initial model for incremental training and evaluation.

[0140] It should be noted that the initial model's training parameters should include at least one of the following: quantity threshold, training ratio, and accuracy threshold.

[0141] It should be noted that the initial model's training parameters also include hard example threshold, bounding box threshold, etc.

[0142] It should be noted that, in the embodiments of this application, the training parameters of the initial model can be set to fixed values, or they can be set to a set of training parameters corresponding to one incremental training.

[0143] It should be noted that this application allows for pre-setting the program to configure the corresponding training parameters for each initial model training iteration. When the current training parameters are met, incremental training of the initial model can be triggered. Alternatively, the training parameters can be manually adjusted for each iteration.

[0144] It should be noted that because object detection models are relatively complex and require a large amount of data, their training time is quite long. If unexpected errors occur during training, causing the training to terminate unexpectedly, all incremental training of the object detection model will be wasted. To address this issue, this embodiment uses model persistence to save temporary data of the object detection model during training, that is, deploying the initial model in ckpt format to the cloud for incremental training of the object detection model in the cloud.

[0145] In step S840 of some embodiments, edge computing is close to end users and geographically dispersed, which can support application services with low latency, location awareness, high mobility and high QoS. Therefore, the second format target detection model, i.e. pb format target detection model, is deployed on the edge node as the deployment model.

[0146] It should be noted that if the object detection model is to be provided to users for object detection, then only the forward propagation process needs to be completed. In this case, model persistence is used to save only the variables in the forward propagation process and fix the variables, that is, the deployment model in pb format is deployed on edge nodes for object detection and inference.

[0147] It's important to note that the deployment model and the initial model are two different formats of the same model. The difference between ckpt and pb persistence methods is that ckpt files separate and save the model structure and weights, facilitating the training process; while pb files are serialized files, facilitating deployment and offline prediction. Therefore, setting the initial model to ckpt format is beneficial for training the object detection model.

[0148] Please see Figure 7 In some embodiments, step S910 may be included, but is not limited to, between step S300 and step S400:

[0149] In step S910, the cloud converts the sample model from the first format to the second format and evaluates the sample model in the second format so as to send the sample model that meets the preset conditions to the edge node.

[0150] In step S910 of some embodiments, this application uses a cloud-edge collaborative approach to train the object detection model. The initial model of the cloud node is incrementally trained by calling the TensorFlow function. In order to improve the training efficiency of the object detection model, the format of the initial model is set to ckpt and the format of the deployment model is set to pb. Therefore, it is necessary to convert the format of the sample model obtained by incremental training from cpkt to pb so as to facilitate the performance evaluation of the sample model and the updating of the deployment model.

[0151] In one embodiment, the target detection model is applied to the detection of reflective clothing and safety helmets in a smart construction site. The detection of reflective clothing and safety helmets is achieved by deploying the target detection model at edge nodes. The target detection model at the edge nodes is in .pb format, also known as the deployment model. This embodiment uses the YOLOv5 algorithm to detect safety helmets and reflective clothing. However, factors such as camera aging can cause changes in the data distribution of the deployment model, making the current AI camera deployment model unsuitable for the current environment, thus causing a decrease in target detection accuracy. Therefore, the deployment model at the edge nodes needs to be readjusted periodically with new data so that the model can continuously adapt to the current sample images. The target detection method provided in this application continuously monitors the difficult sample samples generated by the edge nodes. When the number of difficult sample samples reaches a preset training condition, the initial model is incrementally trained based on the difficult sample samples, and the sample model obtained from the incremental training is evaluated. When the evaluation result of the sample model meets the preset conditions, the deployment model is updated based on the sample model. This method automatically starts training, evaluation, and deployment according to the set training parameters, and can immediately reflect newly generated data in the model. Furthermore, the update process does not take up manual time, the update cycle is shorter, and the model detection accuracy is continuously improved.

[0152] The target detection method provided in this application is applied to a cloud-edge collaborative system. This method is implemented through edge nodes and the cloud in the cloud-edge collaborative system. First, the edge node acquires multiple sample images and infers from the sample images using a pre-set deployment model to obtain difficult sample images. The difficult sample images are then sent to the cloud. Next, the cloud incrementally trains the pre-set initial model based on the difficult sample images to obtain a sample model. The sample model is then evaluated, and sample models whose evaluation results meet preset conditions are sent to the edge node. Then, the edge node updates the deployment model based on the sample model. The updated deployment model is the trained target detection model. Finally, the edge node acquires the image to be detected and inputs the image to be detected into the target detection model to obtain the target detection result. The object detection method provided in this application deploys the object detection model on edge nodes, where the deployed model can be used for object detection. Meanwhile, the model is trained and related data is stored in the cloud, making full use of the computing and storage capabilities of the cloud. Furthermore, deploying the model on edge nodes facilitates offline detection, is not constrained by network conditions, and is more convenient and faster. This model training method can train and learn in a timely manner based on newly generated sample images, shortening the update cycle of the object detection model and further improving the accuracy of object detection.

[0153] Please see Figure 8 This application also provides a target detection device that can implement the above-described target detection method. The device includes:

[0154] Image acquisition module 100 is used to control the edge nodes to acquire multiple sample images;

[0155] The edge inference module 200 is used to control the edge nodes to infer each sample image through a pre-set deployment model, obtain difficult sample examples, and send the difficult sample examples to the cloud.

[0156] The incremental training module 300 is used to control the cloud to perform incremental training on the pre-set initial model based on difficult example samples to obtain the sample model.

[0157] The model evaluation module 400 is used to control the cloud to evaluate the sample models and send the sample models whose evaluation results meet the preset conditions to the edge nodes.

[0158] The model update module 500 is used to control the edge nodes to update the deployed model according to the sample model, and to use the updated deployed model as the trained object detection model.

[0159] The target detection module 600 is used to control the edge nodes to acquire the image to be detected and input the image to be detected into the target detection model to obtain the target detection result.

[0160] The specific implementation of this target detection device is basically the same as the specific implementation of the target detection method described above, and will not be repeated here.

[0161] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described target detection method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0162] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0163] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0164] The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the target detection method of the embodiments of this application.

[0165] The input / output interface 603 is used to implement information input and output;

[0166] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0167] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0168] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0169] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described target detection method.

[0170] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0171] The target detection method, device, electronic equipment, and storage medium provided in this application first acquire multiple sample images. Then, they infer from these sample images using a pre-set deployment model at the edge nodes to obtain difficult sample images, which are then sent to the cloud. Next, an initial model pre-set in the cloud is incrementally trained based on these difficult sample images to obtain a sample model. The sample model is then evaluated by the cloud, and models whose evaluation results meet preset conditions are sent to the edge nodes. Finally, the deployment model is updated based on the sample model, and the updated deployment model is the trained target detection model. This application employs a cloud-edge collaborative approach to train the target detection model, fully utilizing the computing and storage capabilities of the cloud. Newly generated data, i.e., sample images, can be immediately reflected in the model, and the model training and updating process does not consume manual time, shortening the model update cycle and further improving the target detection accuracy of the model.

[0172] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0173] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0176] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0177] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between the devices or units may be through some interfaces, and the indirect coupling or communication connection may be electrical, mechanical, or other forms.

[0179] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A target detection method, characterized in that, Applied to a cloud-edge collaborative system, the cloud-edge collaborative system including edge nodes and a cloud, the method includes: Multiple sample images are obtained from the edge nodes; The edge nodes perform inference on each of the sample images using a pre-set deployment model to obtain difficult sample examples, and then send the difficult sample examples to the cloud. The cloud determines a set of difficult examples based on the difficult example samples and the labeled data corresponding to the difficult example samples, and extracts a preset training ratio of the difficult example samples from the set of difficult examples as a training sample set to incrementally train the pre-set initial model to obtain a sample model; The cloud-based system removes difficult example samples from the training sample set to obtain an evaluation sample set. Each evaluation sample in the evaluation sample set is then input into the sample model to obtain a second sample detection result for each evaluation sample. Based on the labeled data corresponding to each evaluation sample and the second sample detection result, the evaluation accuracy of the sample model is determined. When the evaluation accuracy is greater than or equal to a preset accuracy threshold, the sample model is sent to the edge node. The edge nodes update the deployment model based on the sample model, and the updated deployment model is used as the trained target detection model. The image to be detected is obtained from the edge nodes, and the image to be detected is input into the target detection model to obtain the target detection result.

2. The target detection method according to claim 1, characterized in that, The process of obtaining difficult sample samples by reasoning through each sample image using a pre-set deployment model by the edge nodes includes: For each sample image, the edge nodes infer the sample image using a pre-set deployment model to obtain at least one predicted bounding box and the confidence level corresponding to the predicted bounding box; The difficulty coefficient of the sample image is calculated based on the predicted bounding box and the confidence level corresponding to the predicted bounding box. When the difficulty coefficient is greater than the preset difficulty threshold, the sample image is determined to be a difficult sample.

3. The target detection method according to claim 1, characterized in that, After sending the difficult example sample to the cloud, the method further includes: The cloud platform performs statistical analysis on the received difficult sample samples to obtain the number of difficult sample samples. When the number of difficult examples is greater than or equal to a preset threshold, the process is triggered to obtain the pre-set initial model for incremental training.

4. The target detection method according to claim 3, characterized in that, The process involves the cloud determining a difficult example set based on the difficult example samples and their corresponding labeled data, and extracting a preset training ratio of difficult example samples from the difficult example set as a training sample set to incrementally train a pre-set initial model to obtain a sample model, including: Obtain the labeled data corresponding to each of the aforementioned difficult example samples; A set of difficult examples is determined based on multiple difficult example samples and the corresponding labeled data of the difficult example samples; A training sample set is extracted from the set of difficult examples according to a preset training ratio; Each training sample in the training sample set is input into the initial model to obtain the first sample detection result corresponding to each training sample. Based on the labeled data corresponding to each training sample and the detection result of the first sample, the parameters of the initial model are optimized to obtain the sample model.

5. The target detection method according to any one of claims 1 to 4, characterized in that, Before acquiring multiple sample images from the edge nodes, the method further includes: The target detection model is obtained from the cloud. The cloud platform converts the target detection model into a first format and a second format, respectively. The cloud takes the target detection model in the first format as the initial model and sets the training parameters of the initial model, wherein the training parameters include at least one of the following: quantity threshold, training ratio, and accuracy threshold; The cloud takes the target detection model in the second format as the deployment model and sends the deployment model to the edge node.

6. The target detection method according to claim 5, characterized in that, After the cloud performs incremental training on the pre-set initial model based on the difficult example samples to obtain the sample model, the method further includes: The cloud platform converts the sample model from a first format to a second format and evaluates the sample model in the second format, so as to send the sample model that meets the preset conditions to the edge node.

7. A target detection device, characterized in that, An application is made in a cloud-edge collaborative system, which includes edge nodes and a cloud, wherein the target detection device is connected to both the edge nodes and the cloud, and the device includes: The image acquisition module is used to control the edge nodes to acquire multiple sample images; The edge inference module is used to control the edge nodes to infer each of the sample images through a pre-set deployment model, obtain difficult sample examples, and send the difficult sample examples to the cloud. The incremental training module is used to control the cloud to determine the difficult example set based on the difficult example samples and the labeled data corresponding to the difficult example samples, and to extract the difficult example samples with a preset training ratio from the difficult example set as the training sample set, so as to perform incremental training on the pre-set initial model to obtain the sample model; The model evaluation module controls the cloud to remove difficult example samples from the training sample set to obtain an evaluation sample set; inputs each evaluation sample in the evaluation sample set into the sample model to obtain a second sample detection result corresponding to each evaluation sample; determines the evaluation accuracy of the sample model based on the labeled data corresponding to each evaluation sample and the second sample detection result; and sends the sample model to the edge node when the evaluation accuracy is greater than or equal to a preset accuracy threshold. The model update module is used to control the edge nodes to update the deployment model according to the sample model, and use the updated deployment model as the trained target detection model; The target detection module is used to control the edge nodes to acquire the image to be detected and input the image to be detected into the target detection model to obtain the target detection result.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the target detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the target detection method according to any one of claims 1 to 6.

Images