Target object detection method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a target object detection method and device, equipment and a storage medium, and the method comprises the steps: carrying out the first detection of a target object on obtained video data based on a predetermined first detection model, and determining a key frame of the video data according to a detection result corresponding to the first detection; sending the key frame to a cloud, so that the cloud performs second detection of the target object according to the key frame and a second detection model; wherein the second detection model is updated according to the environmental data when the video data is collected before performing the second detection on the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for detecting a target object. Background Art

[0002] In intelligent transportation systems, video object detection plays a vital role. Currently, a single cloud computing or edge computing solution is usually used to detect target objects in video streams. However, although cloud computing solutions have powerful computing capabilities, they are affected by network latency and cannot achieve near-real-time detection of target objects; while edge computing can provide near-real-time detection, it is limited by computing resources and has difficulty processing complex video streams. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, apparatus, device and storage medium for detecting a target object to solve the problem in the related art that a single cloud computing or edge computing solution is used to perform video object detection, resulting in the inability to simultaneously meet the requirements of complex video stream processing and near-real-time detection of target objects.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for detecting a target object, the method comprising: Based on a predetermined first detection model, perform a first detection of the target object on the acquired video data, and determine a key frame of the video data according to a detection result corresponding to the first detection; The key frame is sent to the cloud so that the cloud performs a second detection of the target object based on the key frame and a second detection model; wherein, before the second detection of the target object is performed, the second detection model is updated according to the environmental data when the video data is collected.

[0005] In a second aspect, an embodiment of the present disclosure provides a device for detecting a target object, the device comprising: A detection module, configured to perform a first detection of a target object on the acquired video data based on a predetermined first detection model, and determine a key frame of the video data according to a detection result corresponding to the first detection; A sending module is used to send the key frame to the cloud so that the cloud performs a second detection of the target object based on the key frame and a second detection model; wherein, before the second detection of the target object is performed, the second detection model is updated according to the environmental data when the video data is collected.

[0006] In a third aspect, an embodiment of the present disclosure provides a target object detection device, comprising: a processor; and a memory configured to store computer executable instructions, wherein the computer executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] At least one of the above technical solutions provided by the embodiments of the present invention can achieve the following technical effects: In an embodiment of the present invention, a first detection of a target object can be performed on the acquired video data based on a predetermined first detection model, and a key frame of the video data can be determined according to a detection result corresponding to the first detection, and the key frame can be sent to the cloud, so that the cloud can perform a second detection of the target object based on the key frame and a second detection model, wherein the second detection model is updated according to the environmental data when the video data is collected before the second detection is performed on the target object.

[0010] It can be seen from the above-mentioned embodiments of the invention that after acquiring the video data, this solution can first perform a first detection of the target object on the video data based on the first detection model, and determine the key frame of the video data according to the detection result corresponding to the first detection, and then send the key frame to the cloud, so that the cloud can perform a second detection of the target object according to the key frame and the second detection model. Therefore, in the embodiments of the present invention, after acquiring the video data, the target object can be first detected in real time on the video data, and the video key frame can be determined according to the detection result of the real-time detection, and the video key frame can be sent to the cloud, so that the cloud can accurately identify the target object according to the second detection model timely updated by the environmental data corresponding to the video data, and the video key frame. Since the target object can be instantly identified after the video data is acquired, the effect of near real-time detection can be well achieved. At the same time, since video key frames can be identified and the target objects can be accurately identified for the video key frames through the cloud, the processing of complex video streams can be well realized, and high-quality target object recognition results can be obtained. Therefore, through collaboration with the cloud, the problem of the related technology that a single cloud computing or edge computing solution is used for video object detection, which cannot simultaneously meet the requirements of complex video stream processing and near real-time detection of target objects, can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor. Figure 1 One of the flow charts of a method for detecting a target object provided by an embodiment of the present invention; Figure 2 A second flow chart of a method for detecting a target object provided by an embodiment of the present invention; Figure 3 A third flow chart of a method for detecting a target object provided by an embodiment of the present invention; Figure 4 A fourth flowchart of a method for detecting a target object provided by an embodiment of the present invention; Figure 5 A fifth flowchart of a method for detecting a target object provided by an embodiment of the present invention; Figure 6 A sixth flowchart of a method for detecting a target object provided by an embodiment of the present invention; Figure 7A seventh flowchart of a method for detecting a target object provided by an embodiment of the present invention; Figure 8 FIG8 is a flow chart of a method for detecting a target object provided by an embodiment of the present invention; Fig. 9 A ninth flowchart of a method for detecting a target object provided by an embodiment of the present invention; Fig.10 A schematic diagram of the module composition of a target object detection device 1000 provided in one embodiment of the present invention; Fig.11 A schematic diagram of the hardware structure of a target object detection device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the technical parties in this technical field better understand the technical aspects of one or more embodiments of the present disclosure, and to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0013] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0014] See also Figure 1 , Figure 1 One of the flow diagrams of a method for detecting a target object provided by an embodiment of the present invention is applied to an edge device, such as an edge device, such as Figure 1 As shown, the method comprises the following steps: Step 102: Based on a predetermined first detection model, a first detection of a target object is performed on the acquired video data, and a key frame of the video data is determined according to a detection result corresponding to the first detection.

[0015] Step 104: Send the key frame to the cloud so that the cloud performs a second detection of the target object based on the key frame and the second detection model; wherein, before performing the second detection on the target object, the second detection model is updated according to the environmental data when the video data is collected.

[0016] In the embodiment of the present invention, the video data may be acquired first. When acquiring the video data, the video data may be acquired through a pre-set sensor, camera, etc., and the embodiment of the present invention does not limit this.

[0017] After the video data is acquired, a first detection of the target object may be performed on the acquired video data based on a predetermined first detection model.

[0018] In one embodiment of the invention, the first detection model may be a model that has been pre-lightweighted and optimized. Specifically, the initial first detection model may be obtained first, and when the model is obtained, it may be created using an existing model creation method. After obtaining the initial first detection model, the initial first detection model is pruned to obtain a pruned detection model, and then the feature fusion network in the pruned detection model is compressed to obtain a first compressed detection model, and the first compressed detection model is converted to a weight numerical format and subjected to knowledge distillation to obtain a second compressed detection model. Finally, the second compressed detection model is algorithmically modified to obtain the first detection model, wherein the algorithm modification may include at least one of the following: convolution operation modification, memory access mode modification.

[0019] In one example, considering that the first detection model is a lightweight optimized detection model, in this example, for the convenience of description, the first detection model is referred to as a lightweight detection framework. When lightweight optimization is performed on the initial first detection model to obtain a lightweight detection framework, in order to reduce the computational burden of the model, pruning technology can be used to optimize the feature extraction network of the lightweight detection framework. Pruning technology can reduce the complexity and computational complexity of the model by removing unimportant weights or neurons in the model. During the pruning process, a weight importance score can be defined. To evaluate each weight For example, each weight can be evaluated based on the product of the absolute value of the gradient of the loss function corresponding to the weight and the probability of feature map activation. The importance of can be calculated by:

[0020] in, is the loss function; and They are the input feature map and the output feature map respectively; is the feature map location The activation probability at the location reflects the importance of the location in the feature map.

[0021] After pruning the initial first detection model, considering that feature fusion is a computationally intensive process in multi-scale prediction, this example uses compression technology to reduce the number of parameters of the fusion network to improve efficiency. For example, by introducing a compression factor , the original feature fusion operation can be simplified to:

[0022] in, is the first feature maps; is the compressed feature map. By integrating feature maps from different scales through the fusion optimization strategy of multi-scale features, it can not only retain important information at each scale, but also reduce redundancy, thereby enhancing the model's sensitivity to changes in target size.

[0023] After compressing the feature fusion network in the pruned model, in order to ensure the normal operation of the first detection model, this example uses model compression technology, including weight quantization and knowledge distillation. Weight quantization can convert floating-point weights into low-precision representations to reduce the model size; while knowledge distillation can transfer the knowledge of a large and complex model to a smaller and simpler model. Model compression can be expressed as:

[0024] in, It is the original model; It is a compressed model; It is a compression operation.

[0025] After the weight numerical format conversion and knowledge distillation processing, the algorithm can be further modified to optimize performance, where the algorithm modification can include improving convolution operations and memory access patterns to reduce computing delays. In addition, hardware can also be accelerated, for example, dedicated deep learning accelerators can be used to increase hardware processing speed.

[0026] See also Figure 2 , is a flowchart of this example, Figure 2 It mainly includes the following parts: 1. Pruned feature extraction network (M1): M1 reduces the computational complexity of the model by removing unimportant weights while maintaining the extraction of key target features.

[0027] 2. Compression of feature fusion network (M2): M2 uses compression technology to reduce the number of network parameters and improve the efficiency of multi-scale prediction.

[0028] 3. Optimization of Multi-Scale Prediction (M3): M3 makes predictions at different levels to capture objects of different sizes.

[0029] 4. Application of model compression technology (M4): M4 applies weight quantization and knowledge distillation technology to reduce the model size and improve the operating efficiency on edge devices.

[0030] 5. Performance optimization of lightweight detection framework (M5): M5 further improves processing speed through algorithm optimization and hardware acceleration.

[0031] exist Figure 2 In the figure, arrows can be used to represent data flow and information transfer in each part: the output of M1 points to M3 through an arrow, indicating the transfer of feature extraction; the output of M2 is transferred to M3 through an arrow, indicating the transfer of feature fusion; the output of M3 points to M4 and M5 respectively, indicating the transfer of multi-scale prediction results; finally, the outputs of M4 and M5 are summarized into the performance optimized detection results.

[0032] It can be seen from the embodiments of the present invention that the first detection model, i.e. the lightweight detection framework in the example, can achieve more efficient and accurate target object detection through its own lightweight optimization, and can also provide key data for subsequent high-precision detection in the cloud. The design of the first detection model fully considers the resource constraints and device heterogeneity in the edge computing environment, and provides a solid foundation for achieving effective near-real-time target object detection.

[0033] In the embodiment of the present invention, after the first detection of the target object is performed on the acquired video data, the key frame of the video data may be determined according to the detection result corresponding to the first detection.

[0034] In one embodiment of the present invention, when performing the first detection of the target object, the first detection model can detect some frames that may have semantic changes, so as to determine the key frames of the video later based on these frames. Whether there is a semantic change can be determined based on the image area where the target object identified and tracked in the video data is located, for example, by the color, texture, shape, and position coordinates of the image area where the target object is located.

[0035] In an embodiment of the present invention, a candidate key frame can be determined based on the detection result corresponding to the first detection, that is, a frame that may have semantic changes relative to its adjacent previous frame in the video data, and then, a target image area where the target object is located is determined from the candidate key frames, and a key frame is determined from the candidate key frames based on the semantic similarity of the target image area between adjacent candidate key frames, wherein the semantic similarity can include at least one of the following: color similarity, texture similarity, and shape similarity.

[0036] In one example, considering that the selection of key frames is crucial to balancing the computational load on the cloud and maintaining detection accuracy, this example proposes an adaptive key frame selection method to dynamically select key frames with significant semantic changes from video data so as to upload them to the cloud for high-precision detection. For example, in order to quantify the semantic changes between video frames, for two consecutive frames and , semantic change measure It can be calculated by the following formula:

[0037] in, is the number of objects detected in the frame; Is an object The similarity measure between two consecutive frames can be calculated based on various features such as color, texture, shape, etc.

[0038] Based on the semantic change metric, this example also designs a keyframe selection strategy that can use an adaptive threshold To decide whether to use the current frame as a key frame. If the semantic change measure exceeds the threshold, the frame is selected as a key frame:

[0039] in, is a Boolean variable indicating the Whether to select a keyframe.

[0040] In order to adapt to different traffic environments and detection requirements, this example also proposes an adaptive threshold adjustment mechanism. This mechanism can dynamically adjust the threshold according to the current network load and detection performance feedback. :

[0041] in, is the detection performance of the current frame; Based on historical best model performance And add adjustable variables The target performance values ​​formed; It is the adjustment step size, which can be a default value or set in advance by relevant personnel.

[0042] See also Figure 3 , is a flowchart of this example, Figure 3 It mainly includes the following parts: 1. Semantic Change Measurement Module (M1): M1 is responsible for quantifying the semantic changes between video frames by calculating the semantic similarity between two consecutive frames.

[0043] 2. Key frame selection strategy module (M2): M2 uses an adaptive threshold to decide whether to upload the current frame as a key frame based on the semantic change measurement results.

[0044] 3. Adaptive threshold adjustment module (M3): M3 dynamically adjusts the threshold according to the current network load and detection performance feedback to adapt to different traffic environments and detection requirements.

[0045] In Figure 3, arrows are used to represent data flow and information transmission in each part. The measurement results of M1 drive M2 to select key frames and guide M3 to dynamically adjust the threshold; the selection results of M2 are fed back to M3 to optimize the threshold adjustment strategy; the adjustment results of M3 affect the key frame selection efficiency of M2. Finally, the outputs of all modules are summarized at the end point B, indicating the completion of the entire adaptive key frame selection process.

[0046] In the embodiment of the present invention, video key frames can be dynamically selected in different scenarios, which effectively reduces the computing burden of the cloud and ensures the accuracy of the first detection.

[0047] After determining the key frames of the video data, the key frames may be sent to the cloud, so that the cloud performs a second detection of the target object based on the key frames and the second detection model.

[0048] In one embodiment of the present invention, key frames can be sent to the cloud so that the cloud allocates processing resources according to the number and type of key frames, and based on the allocated processing resources, performs a second detection of the target object according to the key frames and the second detection model.

[0049] In the embodiment of the present invention, the cooperation mechanism between the edge device and the cloud is the core, which realizes the close cooperation between the edge device and the cloud through efficient data transmission and computing offloading strategy. This mechanism not only improves the real-time performance of video object detection, but also ensures the high-precision completion of the detection task.

[0050] In one example, when the edge device determines the frame When it is a key frame, the transmission priority can be calculated using the following formula :

[0051] in, and is the weight coefficient; is a measure of semantic change; It is an indicator of the current network quality, ensuring that key frames are transmitted first when network conditions are good.

[0052] After the cloud receives the key frames, it dynamically adjusts the resource allocation according to the number and type of key frames received. You can define the resource allocation function , according to the feature complexity of the key frame and the current length of the queue of frames to be processed on the cloud To allocate resources:

[0053] This ensures the rational allocation of computing resources and optimizes processing efficiency.

[0054] While the edge device sends key frames to the cloud and waits for the cloud to feedback the detection results, it can use the lightweight detection framework, that is, the first detection model, to perform real-time detection. This real-time detection can quickly respond to changes in the traffic environment and provide feedback information to the cloud. Suppose the edge device detection feedback cycle is , then the feedback frequency It can be expressed as:

[0055] See also Figure 4 , is a flowchart of this example, Figure 4 It mainly includes the following parts: 1. Key frame transmission strategy module (M1): M1 is responsible for calculating the transmission priority to ensure that key frames are transmitted first when network conditions are good.

[0056] 2. Cloud model dynamic loading and allocation module (M2): M2 dynamically adjusts resource allocation according to the number and type of key frames received to optimize processing efficiency.

[0057] 3. Edge device real-time detection and feedback module (M3): M3 uses a lightweight detection framework for real-time detection, quickly responds to changes in the traffic environment, and provides feedback information to the cloud.

[0058] In Figure 4, arrows are used to represent data flows and information transmission in each part. The transmission strategy of M1 drives M2 to dynamically load and allocate resources, and promotes M3 to perform real-time detection and feedback; M2's resource allocation supports the efficient execution of M1 and M3; M3's real-time detection and feedback optimizes M1's transmission strategy and M2's resource allocation. Finally, the outputs of all modules are summarized at the end point B, indicating the completion of the entire edge cloud computing collaboration process.

[0059] The embodiments of the present invention can achieve close cooperation between edge devices and the cloud through efficient data transmission and computing offloading strategies.

[0060] In one embodiment of the present invention, after the edge device performs a first detection of the target object on the acquired video data, it can also obtain the environmental data when the video data is collected; wherein the environmental data may include at least one of the following: weather, time, and traffic scene type, and the traffic scene type may include urban roads, highways, rural roads, etc. Then, the acquired environmental data can be sent to the cloud so that the cloud updates the second detection model according to the environmental data.

[0061] In this embodiment, when the edge device acquires video data, it can also acquire the environmental data at that time. When acquiring, it can be acquired through the set sensors or communication systems. The implementation of the present invention does not impose any restrictions on this.

[0062] In one example, after acquiring the environmental data, the context information of the target object's environmental data can be analyzed and fused, and represented in the form of a feature vector:

[0063] in, It is a feature vector of context information, which can represent different types of environmental data; It is The value of a context feature.

[0064] According to the obtained context information, the second detection model in the cloud is adaptively adjusted to optimize the performance of the second detection model in a specific environment, wherein the adaptive adjustment can be achieved by fine-tuning parameters, modifying the structure, or completely retraining. The model adjustment can be expressed as:

[0065] in, are the model parameters after further adjustment; It is the latest cloud model parameters just aggregated; It is the parameter adjustment amount calculated based on the context information. This operation is triggered after the cloud model is iteratively updated.

[0066] The embodiment of the present invention also introduces a performance evaluation mechanism, which is achieved by comparing the recall rate index before and after the fusion of context information:

[0067] in, is the amount of context-aware performance improvement; is the detection performance after integrating context information; is the detection performance without integrating context information.

[0068] See also Figure 5 , is a flowchart of this example, Figure 5 It mainly includes the following parts: 1. Feedback mechanism establishment module (M1): M1 is responsible for receiving key frame detection results and context information fed back by edge devices, and providing data support for updating the second detection model in the cloud.

[0069] 2. Model performance dynamic evaluation module (M2): M2 evaluates the performance of the second detection model in the cloud based on feedback data to ensure that the direction of model optimization is consistent with actual needs.

[0070] 3. Continuous learning strategy module (M3): M3 implements incremental learning of cloud models, allowing the model to retain memory of old data while continuously learning new data.

[0071] 4. Model adaptive adjustment module (M4): M4 adaptively adjusts model parameters according to the evaluation results and learning strategies to achieve continuous optimization of the second detection model.

[0072] exist Figure 5 In the figure, arrows are used to represent data flow and information transmission. The feature representation results of M1 are input to the cloud model adjustment module of M2 to guide the adjustment of model parameters; the adjustment results of M2 are fed back to the performance evaluation module of M3 to trigger the performance evaluation process; the evaluation results of M3 are fed back to M2 to adjust the model adjustment strategy. Finally, the outputs of all modules are summarized at the end point B, indicating the completion of the entire context-aware technology process.

[0073] The embodiment of the present invention takes into account the diversity and real-time nature of environmental changes, and can enable the second detection model to better adapt to different environments, thereby improving the accuracy and robustness of detection.

[0074] In an embodiment of the present invention, when the acquired environmental data is sent to the cloud so that the cloud updates the second detection model according to the environmental data, the detection result corresponding to the first detection and the acquired environmental data can be sent to the cloud so that the cloud updates the second detection model according to the detection result corresponding to the first detection and the environmental data.

[0075] In one example, the second detection model in the cloud can be updated, and performance evaluation and parameter optimization can be performed based on the detection results fed back by the edge device and the above context information. , then the update of the cloud model can be expressed as:

[0076] in, It is an updated model; It is the current cloud model; is the amount of model parameter adjustment based on feedback data The performance evaluation of the second cloud-based detection model is achieved by calculating the detection recall rate indicator. Suppose the performance evaluation function is , then the performance evaluation of the cloud model after receiving the new key frame can be expressed as:

[0077] in, is the performance evaluation value of the updated model; It is a new key frame dataset.

[0078] In order to achieve continuous learning of cloud models, an incremental learning strategy can be adopted. Incremental learning allows the model to retain the memory of old data while continuously learning new data. The incremental learning strategy can be implemented by the following formula:

[0079] in, It is the update sign; is the threshold at which performance degrades.

[0080] See also Figure 6 , is a flowchart of this example, Figure 6 It mainly includes the following parts: 1. Feedback mechanism establishment module (M1): M1 is responsible for receiving key frame detection results and context information fed back by edge devices, providing data support for updating the cloud model.

[0081] 2. Model performance dynamic evaluation module (M2): M2 evaluates the performance of the cloud model based on feedback data to ensure that the direction of model optimization is consistent with actual needs.

[0082] 3. Continuous learning strategy module (M3): M3 implements incremental learning of cloud models, allowing the model to retain memory of old data while continuously learning new data.

[0083] 4. Model adaptive adjustment module (M4): M4 adaptively adjusts model parameters according to evaluation results and learning strategies to achieve continuous optimization of the model.

[0084] exist Figure 6 In the diagram, arrows are used to represent data flow and information transmission. The feedback data collected by M1 drives M2 to evaluate model performance, triggers M3's continuous learning strategy, and guides M4 to perform model adaptive adjustment. The evaluation results of M2 are fed back to M3 and M4 to optimize the learning strategy and adjustment effect. M3's learning strategy affects the adjustment direction and amplitude of M4. Finally, the outputs of all modules are summarized at the end point B, indicating the completion of the entire cloud model optimization and continuous learning process.

[0085] In an embodiment of the present invention, the cloud model adaptively adjusts the model parameters based on the detection results and context information fed back by the edge device, continuously iterates the cloud model, and updates the model parameters to adapt to the continuous changes of the model in the environment.

[0086] Through the above-mentioned cloud model optimization and continuous learning module, the embodiment of the present invention realizes dynamic adjustment and optimization of model parameters, improves the accuracy and robustness of the detection model in different environments, and provides strong technical support for intelligent transportation systems.

[0087] From the above content, it can be seen that the embodiments of the present invention can realize the detection of target objects through the cooperation between the edge device and the cloud. Specifically, preliminary lightweight detection can be performed on the edge device, and the key frames can be transmitted to the cloud for high-precision detection to achieve a balance between real-time performance and accuracy.

[0088] See also Figure 7 , is a schematic diagram of the overall process of the edge device and the cloud collaboration in the implementation of the present invention, which mainly includes the following parts: 1. Lightweight Detection Framework (M1): M1 is deployed on edge devices for preliminary processing of real-time agricultural video streams, including target tracking and preliminary detection.

[0089] 2. Adaptive keyframe selection (M2): M2 is responsible for dynamically selecting keyframes from the video stream, which will be uploaded to the cloud for deeper analysis.

[0090] 3. Edge cloud computing collaboration mechanism (M3): M3 involves data transmission and task collaboration between edge devices and the cloud to ensure efficient data transmission and processing.

[0091] 4. Context-aware technology (M4): M4 uses contextual information of the environment, such as weather, time, traffic scenes, etc., to adjust the detection model in the cloud.

[0092] 5. Cloud model optimization and continuous learning (M5): M5 means that the cloud model continuously optimizes itself based on real-time feedback to adapt to the ever-changing traffic environment.

[0093] 6. Result feedback and deployment (M6): M6 is responsible for feeding back the results of cloud processing to edge devices and updating the model of edge devices when necessary.

[0094] exist Figure 7 In the figure, arrows are used to represent data flow and information transmission. The output of M1 points to M2 through an arrow, indicating the transmission of preliminary detection results; the output of M2 selectively points to M3, indicating the upload of key frames; M3 interacts with M1 and M2 at the same time to handle data transmission and task offloading; the output of M4 points to M5 through an arrow, indicating the transmission of context information; the output of M5 points to M6 through an arrow, indicating the deployment of the optimized model; M6 also receives feedback from the cloud, forming a closed-loop optimization.

[0095] The following two examples illustrate the collaboration process between edge devices and the cloud: See also Figure 8 , is an example flow chart of an autonomous agricultural machine scenario, which is used to detect and analyze the surrounding environment of the agricultural machine in real time. The specific process is as follows: A. Sensor data collection: Autonomous agricultural machinery collects images and contextual information of the surrounding environment in real time through the sensors it carries.

[0096] B. Lightweight detection framework (M1): A lightweight detection framework is run on the edge computing unit of the vehicle to perform preliminary processing on the sensor data.

[0097] C. Adaptive keyframe selection (M2): Based on the contextual information of the vehicle, keyframes are dynamically selected and uploaded to the cloud for high-precision detection.

[0098] D. Edge cloud computing collaboration (M3): The cloud server receives key frame data and collaborates with edge devices to complete environmental perception tasks.

[0099] E. Context-aware (M4): The cloud server uses the vehicle’s contextual information to adjust the detection model to adapt to different driving environments.

[0100] F. Cloud-based model optimization and learning (M5): The cloud server continuously optimizes the detection model based on real-time feedback and sends updates through edge devices.

[0101] G. Real-time environmental perception: Edge devices use optimized models to perform real-time environmental perception to assist in autonomous driving decision-making.

[0102] It can be seen that the embodiments of the present invention can be well applied in the scenario of autonomous driving agricultural machinery, and the detection of target objects can be achieved through the collaboration of edge devices and the cloud to assist autonomous driving decision-making.

[0103] See also Fig. 9 , is an example flow chart for traffic violation monitoring scenario, which is used to automatically detect and record traffic violations. The specific process is as follows: A. Violation data collection: Traffic surveillance cameras capture images of possible violations in real time.

[0104] B. Keyframe screening: The edge computing unit uses an adaptive keyframe selection method (M1) to screen out keyframes with potential violations.

[0105] C. Preliminary Violation Identification: The edge computing unit uses a lightweight detection framework (M2) to perform preliminary violation identification.

[0106] D. Upload violation data: upload the selected key frames to the cloud server.

[0107] E. Cloud-based violation identification: The cloud server uses the cloud-based model optimization and continuous learning module (M3) to accurately identify violations.

[0108] F. Contextual information fusion: The cloud server applies context awareness (M4) to combine traffic rules and scene context to improve recognition accuracy.

[0109] G. Violation records: The cloud server records confirmed violations and generates violation reports.

[0110] H. Violation handling: Send the violation report to the traffic management department for subsequent violation handling and notification.

[0111] I. System performance feedback: The traffic management department sends the processing results and feedback information back to the cloud server for continuous learning and optimization of the model.

[0112] It can be seen that the embodiments of the present invention can be well applied in the scenario of traffic violation monitoring, and the detection of traffic violations can be achieved through the collaboration of edge devices and the cloud.

[0113] In an embodiment of the present invention, a first detection of a target object can be performed on the acquired video data based on a predetermined first detection model, and a key frame of the video data can be determined according to a detection result corresponding to the first detection, and the key frame can be sent to the cloud, so that the cloud can perform a second detection of the target object based on the key frame and a second detection model, wherein the second detection model is updated according to the environmental data when the video data is collected before the second detection is performed on the target object.

[0114] It can be seen from the above-mentioned embodiments of the invention that after acquiring the video data, this solution can first perform a first detection of the target object on the video data based on the first detection model, and determine the key frame of the video data according to the detection result corresponding to the first detection, and then send the key frame to the cloud, so that the cloud can perform a second detection of the target object according to the key frame and the second detection model. Therefore, in the embodiments of the present invention, after acquiring the video data, the target object can be first detected in real time on the video data, and the video key frame can be determined according to the detection result of the real-time detection, and the video key frame can be sent to the cloud, so that the cloud can accurately identify the target object according to the second detection model timely updated by the environmental data corresponding to the video data, and the video key frame. Since the target object can be instantly identified after the video data is acquired, the effect of near real-time detection can be well achieved. At the same time, since video key frames can be identified and the target objects can be accurately identified for the video key frames through the cloud, the processing of complex video streams can be well realized, and high-quality target object recognition results can be obtained. Therefore, through collaboration with the cloud, the problem of the related technology that a single cloud computing or edge computing solution is used for video object detection, which cannot simultaneously meet the requirements of complex video stream processing and near real-time detection of target objects, can be effectively solved.

[0115] Corresponding to the above-mentioned target object detection method, an embodiment of the present invention further provides a target object detection device, Fig.10 A schematic diagram of the module composition of a target object detection device 1000 provided in an embodiment of the present invention is shown in FIG. Fig.10 As shown, the target object detection device 1000 includes: The detection module 1001 is used to perform a first detection of a target object on the acquired video data based on a predetermined first detection model, and determine a key frame of the video data according to a detection result corresponding to the first detection; The sending module 1002 is used to send the key frame to the cloud so that the cloud performs a second detection of the target object based on the key frame and the second detection model; wherein, before the second detection of the target object is performed, the second detection model is updated according to the environmental data when the video data is collected.

[0116] Optionally, the device further comprises ( Fig.10 not shown): A first acquisition module 1003 is used to acquire an initial first detection model before performing a first detection of a target object on the acquired video data based on a predetermined first detection model; A second acquisition module 1004 is used to perform pruning processing on the initial first detection model to obtain a pruned detection model; A third acquisition module 1005 is used to compress the feature fusion network in the pruning detection model to obtain a first compressed detection model; A first compression module 1006, configured to perform weight value format conversion and knowledge distillation processing on the first compression detection model to obtain a second compression detection model; The first modification module 1007 is used to perform algorithm modification on the second compression detection model to obtain a first detection model; wherein the algorithm modification includes at least one of the following: convolution operation modification and memory access mode modification.

[0117] Optionally, the detection module 1001 is used to: Determining a candidate key frame according to a detection result corresponding to the first detection; Determine a target image region where the target object is located from the candidate key frames; The key frame is determined from the candidate key frames according to the semantic similarity of the target image area between adjacent candidate key frames; wherein the semantic similarity includes at least one of the following: color similarity, texture similarity, and shape similarity.

[0118] Optionally, the sending module 1002 is used to: The key frames are sent to the cloud so that the cloud allocates processing resources according to the number and type of the key frames, and based on the allocated processing resources, performs a second detection of the target object according to the key frames and a second detection model.

[0119] Optionally, the device further comprises ( Fig.10 not shown): The fourth acquisition module 1008 is used to acquire the environmental data when the video data is collected; wherein the environmental data includes at least one of the following: weather, time, and traffic scene type; The fifth acquisition module 1009 is used to send the acquired environmental data to the cloud, so that the cloud updates the second detection model according to the environmental data.

[0120] Optionally, the fifth acquisition module 1009 is used to: The detection result corresponding to the first detection and the acquired environmental data are sent to the cloud, so that the cloud updates the second detection model according to the detection result corresponding to the first detection and the environmental data.

[0121] In an embodiment of the present invention, a first detection of a target object can be performed on the acquired video data based on a predetermined first detection model, and a key frame of the video data can be determined according to a detection result corresponding to the first detection, and the key frame can be sent to the cloud, so that the cloud can perform a second detection of the target object based on the key frame and a second detection model, wherein the second detection model is updated according to the environmental data when the video data is collected before the second detection is performed on the target object.

[0122] It can be seen from the above-mentioned embodiments of the invention that after acquiring the video data, this solution can first perform a first detection of the target object on the video data based on the first detection model, and determine the key frame of the video data according to the detection result corresponding to the first detection, and then send the key frame to the cloud, so that the cloud can perform a second detection of the target object according to the key frame and the second detection model. Therefore, in the embodiments of the present invention, after acquiring the video data, the target object can be first detected in real time on the video data, and the video key frame can be determined according to the detection result of the real-time detection, and the video key frame can be sent to the cloud, so that the cloud can accurately identify the target object according to the second detection model timely updated by the environmental data corresponding to the video data, and the video key frame. Since the target object can be instantly identified after the video data is acquired, the effect of near real-time detection can be well achieved. At the same time, since video key frames can be identified and the target objects can be accurately identified for the video key frames through the cloud, the processing of complex video streams can be well realized, and high-quality target object recognition results can be obtained. Therefore, through collaboration with the cloud, the problem of the related technology that a single cloud computing or edge computing solution is used for video object detection, which cannot simultaneously meet the requirements of complex video stream processing and near real-time detection of target objects, can be effectively solved.

[0123] Corresponding to the above-mentioned target object detection method, an embodiment of the present invention further provides a target object detection device, Fig.11 A schematic diagram of the hardware structure of a target object detection device provided by one embodiment of the present invention.

[0124] The target object detection device may be a terminal device or a server for detecting the target object provided in the above embodiments.

[0125] The detection device of the target object may have relatively large differences due to different configurations or performances, and may include one or more processors 1101 and memory 1102, and one or more storage applications or data may be stored in the memory 1102. Among them, the memory 1102 may be a short-term storage or a permanent storage. The application stored in the memory 1102 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the detection device of the target object. Further, the processor 1101 may be configured to communicate with the memory 1102, and execute a series of computer executable instructions in the memory 1102 on the detection device of the target object. The detection device of the target object may also include one or more power supplies 1103, one or more wired or wireless network interfaces 1104, one or more input and output interfaces 1105, and one or more keyboards 1106.

[0126] Specifically in this embodiment, the detection device of the target object includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the detection device of the target object, and is configured to be executed by one or more processors.

[0127] In an embodiment of the present invention, a first detection of a target object can be performed on the acquired video data based on a predetermined first detection model, and a key frame of the video data can be determined according to a detection result corresponding to the first detection, and the key frame can be sent to the cloud, so that the cloud can perform a second detection of the target object based on the key frame and a second detection model, wherein the second detection model is updated according to the environmental data when the video data is collected before the second detection is performed on the target object.

[0128] It can be seen from the above-mentioned embodiments of the invention that after acquiring the video data, this solution can first perform a first detection of the target object on the video data based on the first detection model, and determine the key frame of the video data according to the detection result corresponding to the first detection, and then send the key frame to the cloud, so that the cloud can perform a second detection of the target object according to the key frame and the second detection model. Therefore, in the embodiments of the present invention, after acquiring the video data, the target object can be first detected in real time on the video data, and the video key frame can be determined according to the detection result of the real-time detection, and the video key frame can be sent to the cloud, so that the cloud can accurately identify the target object according to the second detection model timely updated by the environmental data corresponding to the video data, and the video key frame. Since the target object can be instantly identified after the video data is acquired, the effect of near real-time detection can be well achieved. At the same time, since video key frames can be identified and the target objects can be accurately identified for the video key frames through the cloud, the processing of complex video streams can be well realized, and high-quality target object recognition results can be obtained. Therefore, through collaboration with the cloud, the problem of the related technology that a single cloud computing or edge computing solution is used for video object detection, which cannot simultaneously meet the requirements of complex video stream processing and near real-time detection of target objects, can be effectively solved.

[0129] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the above process when executed by a processor.

[0130] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned target object detection method embodiment and achieve the same effect and function, which will not be repeated here.

[0131] Another embodiment of the present disclosure further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above process is implemented.

[0132] The computer program product in the embodiments of the present disclosure can implement the various processes of the above-mentioned target object detection method embodiment and achieve the same effects and functions, which will not be repeated here.

[0133] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0134] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0135] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0136] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0137] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0138] It should be understood by those skilled in the art that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0143] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0144] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A method for detecting a target object, characterized in that: The method comprises: Based on a predetermined first detection model, perform a first detection of the target object on the acquired video data, and determine a key frame of the video data according to a detection result corresponding to the first detection; The key frame is sent to the cloud so that the cloud performs a second detection of the target object based on the key frame and a second detection model; wherein, before the second detection of the target object is performed, the second detection model is updated according to the environmental data when the video data is collected.

2. The method according to claim 1, characterized in that Before performing a first detection of the target object on the acquired video data based on the predetermined first detection model, the method further includes: Obtaining an initial first detection model; Performing pruning processing on the initial first detection model to obtain a pruned detection model; Compressing the feature fusion network in the pruning detection model to obtain a first compressed detection model; Performing weight numerical format conversion and knowledge distillation processing on the first compression detection model to obtain a second compression detection model; Performing algorithmic modification on the second compression detection model to obtain a first detection model; wherein the algorithmic modification includes at least one of the following: convolution operation modification, memory access mode modification.

3. The method according to claim 1, characterized in that The step of determining the key frame of the video data according to the detection result corresponding to the first detection includes: Determining a candidate key frame according to a detection result corresponding to the first detection; Determine a target image region where the target object is located from the candidate key frames; The key frame is determined from the candidate key frames according to the semantic similarity of the target image area between adjacent candidate key frames; wherein the semantic similarity includes at least one of the following: color similarity, texture similarity, and shape similarity.

4. The method according to claim 1, characterized in that: The sending the key frame to the cloud, so that the cloud performs a second detection of the target object according to the key frame and the second detection model, includes: The key frames are sent to the cloud so that the cloud allocates processing resources according to the number and type of the key frames, and based on the allocated processing resources, performs a second detection of the target object according to the key frames and a second detection model.

5. The method according to claim 1, characterized in that After the first detection of the target object is performed on the acquired video data, the method further includes: Acquire environmental data when the video data is collected; wherein the environmental data includes at least one of the following: weather, time, and traffic scene type; The acquired environmental data is sent to the cloud, so that the cloud updates the second detection model according to the environmental data.

6. The method according to claim 5, characterized in that The step of sending the acquired environmental data to the cloud, so that the cloud updates the second detection model according to the environmental data, includes: The detection result corresponding to the first detection and the acquired environmental data are sent to the cloud, so that the cloud updates the second detection model according to the detection result corresponding to the first detection and the environmental data.

7. A detection device for a target object, characterized in that: The device comprises: A detection module, configured to perform a first detection of a target object on the acquired video data based on a predetermined first detection model, and determine a key frame of the video data according to a detection result corresponding to the first detection; A sending module is used to send the key frame to the cloud so that the cloud performs a second detection of the target object based on the key frame and a second detection model; wherein, before the second detection of the target object is performed, the second detection model is updated according to the environmental data when the video data is collected.

8. A target object detection device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images