A target detection method and device, electronic equipment and storage medium

Abstract

This application relates to the field of artificial intelligence technology, providing a target detection method, apparatus, electronic device, and storage medium. It detects targets in video frames, generates target bounding boxes, and then checks if the target bounding boxes exist in preceding and following video frames. If not, it generates first detected bounding boxes in the preceding and following frames and calculates their correlation with the real target bounding boxes. If the correlation indicates that the current video frame is a hard example, it adds the video frame to the hard example set. The hard example set is sent to a server to obtain model update data. After obtaining the update data, the model is updated, and the updated model is used for video stream detection. According to this application, the target detection model's ability to identify targets and its robustness in identifying targets in environments with similar objects can be improved. This application can be applied to financial scenarios to improve the reliability of financial security.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target detection method and apparatus, electronic device and storage medium, which can be applied to financial scenarios. Background Technology

[0002] With the development of artificial intelligence (AI) technology, its application in financial scenarios is becoming increasingly widespread, and the traditional financial industry is gradually transforming into Fintech. Object detection is a crucial application of AI technology. In the security field, object detection helps monitoring systems promptly detect anomalies and provide effective early warnings and alarms. The financial industry, including banks, post offices, and securities firms, is also a key area for security work. Security is an essential daily task in the financial industry, playing a vital role in the normal operation of businesses and achieving sound economic and social benefits.

[0003] Deploying object detection models on lightweight edge devices allows these devices to perform object detection based on pixel-level data in images using relevant algorithms. When two objects share similar pixel clusters, existing object detection techniques may misidentify pseudo-objects that resemble the target. In other words, the object detection model on an edge device is affected by similar objects. When the identified content contains objects similar to the target, the object detection model becomes unstable and prone to false detections. This introduces unreliability into security monitoring in the financial industry. Summary of the Invention

[0004] The main objective of this application is to propose a target detection method, device, electronic device, and storage medium, which aims to improve the model's ability to identify targets and its robustness in identifying targets in environments with similar objects, thereby improving the reliability of security monitoring in the financial industry.

[0005] To achieve the above objectives, a first aspect of this application proposes a target detection method applied to an edge device, the method comprising:

[0006] Video streams are acquired through camera units;

[0007] The target detection model located on the edge device is used to perform target detection on each video frame in the video stream to generate target bounding boxes in each video frame.

[0008] Traverse each video frame in the video stream and perform the following processing on the currently traversed video frame: If there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frame, generate first detection bounding boxes that match the suspected false target bounding boxes in the first M video frames and the last N video frames of the currently traversed video frame, respectively. Calculate the correlation degree based on the first detection bounding boxes and the real target bounding boxes. If the correlation degree determines that the currently traversed video frame belongs to a hard case, add the currently traversed video frame to the hard case set.

[0009] The set of difficult examples is sent to the server so that the server can train a copy of the object detection model based on the set of difficult examples and obtain model update data.

[0010] Receive model update data sent by the server;

[0011] The target detection model is updated based on the model update data to obtain the updated target detection model;

[0012] The updated target detection model is used to perform target detection on the video stream acquired by the camera unit.

[0013] In some possible embodiments of this application, the process of determining whether there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frames includes:

[0014] Obtain the target bounding boxes present in the currently traversed video frames;

[0015] For each acquired target bounding box, determine whether the target bounding box exists in adjacent video frames;

[0016] If the target bounding box exists in the adjacent video frame, then the target bounding box is determined as the real target bounding box;

[0017] If the target bounding box does not exist in the adjacent video frame, then the target bounding box is determined as the suspected false target bounding box.

[0018] In some possible embodiments of this application, before calculating the correlation degree based on the first detected bounding box and the true target bounding box, the method further includes:

[0019] For each of the first M video frames and the last N video frames, the first detection bounding box is enlarged, and a second detection bounding box is generated in the video frame;

[0020] The normalized cross-correlation (NCC) value is calculated based on the second detected bounding box and the suspected false target bounding box. If the NCC value is greater than or equal to the first preset threshold, the video frame is selected as a reserved video frame.

[0021] In some possible embodiments of this application, the calculation of the correlation degree based on the first detected bounding box and the real target bounding box includes:

[0022] For the retained video frame, calculate the intersection-union ratio (IUU) between the first detected bounding box and the real target bounding box in the retained video frame;

[0023] The correlation between the first detected bounding box and the real target bounding box is determined based on the intersection-union ratio.

[0024] In some possible embodiments of this application, generating a first detection bounding box that matches the suspected false target bounding box in the first M video frames and the last N video frames of the currently traversed video frame includes:

[0025] For each of the first M video frames and the last N video frames, a template matching algorithm is used to find the region in the video frame that matches the bounding box of the suspected false target.

[0026] The first detection bounding box is determined based on the found region.

[0027] In some possible embodiments of this application, the step of training a copy of the object detection model based on the set of difficult examples to obtain model update data includes:

[0028] The teacher model is used to label the video frames in the difficult example set to obtain the labeled target bounding boxes corresponding to the video frames;

[0029] The video frames in the set of difficult examples are input into a copy of the target detection model to obtain the predicted target bounding box;

[0030] The loss value is determined based on the labeled target bounding box and the predicted target bounding box corresponding to each video frame in the difficult example set;

[0031] The model parameters of the copy of the target detection model are adjusted according to the loss value until the preset training termination condition is met.

[0032] In some possible embodiments of this application, after receiving the model update data sent by the server and before updating the target detection model according to the model update data, the method further includes:

[0033] Obtain the target detection backup model;

[0034] The target detection backup model is updated based on the model update data to obtain the updated target detection backup model.

[0035] The updated target detection backup model is used to perform target detection on the video stream acquired by the camera unit.

[0036] To achieve the above objectives, a second aspect of this application provides a target detection device, the device comprising:

[0037] The video stream acquisition module is used to acquire video streams through the camera unit;

[0038] The first target detection module is used to perform target detection on each video frame in the video stream using the target detection model located on the edge device, so as to generate target bounding boxes in each video frame;

[0039] The difficult example mining module is used to traverse each video frame in the video stream and perform the following processing on the currently traversed video frame: if there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frame, generate first detection bounding boxes that match the suspected false target bounding boxes in the first M video frames and the last N video frames of the currently traversed video frame, calculate the correlation degree based on the first detection bounding boxes and the real target bounding boxes, and if it is determined that the currently traversed video frame belongs to a difficult example based on the correlation degree, add the currently traversed video frame to the difficult example set, where M is greater than or equal to 0 and N is greater than or equal to 0;

[0040] The sending module is used to send the set of difficult examples to the server so that the server can train a copy of the target detection model based on the set of difficult examples and obtain model update data.

[0041] A receiving module is used to receive model update data sent by the server;

[0042] The update module is used to update the target detection model according to the model update data to obtain the updated target detection model;

[0043] The second target detection module is used to perform target detection on the video stream acquired by the camera unit using the updated target detection model.

[0044] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0046] This application proposes a target detection method, apparatus, electronic device, and storage medium. It detects targets in video frames of an acquired video stream, generates target bounding boxes, and then checks if the target bounding boxes exist in preceding and following video frames. If not, it generates first detected bounding boxes in the preceding and following frames and calculates their correlation with the real target bounding boxes. If the correlation indicates the current video frame is a hard example, it adds the current video frame to the hard example set. The hard example set is sent to a server to obtain model update data. After receiving the update data, the model is updated, and the updated model is used for video stream detection. By detecting unstable target bounding boxes, the detection error in the current model is obtained. The error is sent to the server to retrain the model to obtain updated data. The updated data is used to update the current model, improving the model's ability to recognize targets and its robustness in recognizing targets in environments with similar objects, thereby improving the reliability of security monitoring in the financial industry. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the steps of a target detection method provided in an embodiment of this application;

[0048] Figure 2 yes Figure 1 A schematic diagram of the sub-steps in step S103;

[0049] Figure 3 This is a schematic diagram of the detection process provided in the embodiments of this application.

[0050] Figure 4 yes Figure 1 A schematic diagram of another sub-step in step S103;

[0051] Figure 5 yes Figure 1 A schematic diagram of another sub-step in step S103;

[0052] Figure 6 yes Figure 1 A schematic diagram of another sub-step in step S103;

[0053] Figure 7 yes Figure 1 A schematic diagram of the sub-steps in step S106;

[0054] Figure 8 This application also provides a schematic diagram of the structure of a target detection device.

[0055] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Normalized cross correlation (NCC): A common cross correlation calculation method in template matching, its result describes the correlation between two vectors, windows, or samples of the same dimension. Its value ranges from -1 to 1, where -1 represents that the two vectors are uncorrelated, and 1 represents that the two vectors are correlated. The larger the NCC value, the more similar the two vectors are, and vice versa.

[0062] Intersection over Union (IOU) is a metric used in object detection to measure the accuracy of detecting objects in a given dataset. It represents the overlap ratio between the generated candidate bounding boxes and the original ground truth bounding boxes. This metric measures the correlation between the true and predicted bounding boxes; a higher correlation results in a higher IOU. Ideally, the boxes should overlap completely, with a ratio of 1.

[0063] Template matching is an advanced computer vision technique that identifies portions of an image that match a predefined template. It involves moving the template across the entire image and calculating the similarity between the template and an overlaid window in the image. Template matching is achieved through two-dimensional convolution. In convolution, the value of the output pixel is calculated by multiplying the elements of two matrices and summing the results. One matrix represents the image itself, and the other is the template, which serves as the convolution kernel. The principle is simple: by providing a basic template, similar templates are found in the source image through comparison. The matching process involves comparing the source image in the dataset sequentially with templates in a database using indices. During the matching process, the comparison results are stored in a matrix, and a scoring algorithm is used to determine the optimal estimated position of the template in the dataset.

[0064] With the development of artificial intelligence (AI) technology, its application in financial scenarios is becoming increasingly widespread, and the traditional financial industry is gradually transforming into Fintech. Object detection is a crucial application of AI technology. In the security field, object detection helps monitoring systems promptly detect anomalies and provide effective early warnings and alarms. The financial industry, including banks, post offices, and securities firms, is also a key area for security work. Security is an essential daily task in the financial industry, playing a vital role in the normal operation of businesses and achieving sound economic and social benefits.

[0065] Deploying object detection models on lightweight edge devices allows these devices to perform object detection based on pixel-level data in images using relevant algorithms. When two objects share similar pixel clusters, existing object detection techniques may misidentify pseudo-objects that resemble the target. In other words, the object detection model on an edge device is affected by similar objects. When the identified content contains objects similar to the target, the object detection model becomes unstable and prone to false detections. This introduces unreliability into security monitoring in the financial industry.

[0066] Based on this, embodiments of this application provide a target detection method and apparatus, electronic device and storage medium, which aim to improve the model's ability to identify targets and the robustness of the model when identifying targets in an environment where similar objects exist.

[0067] This application provides a target detection method, apparatus, electronic device, and storage medium, which are specifically described through the following embodiments. First, a target detection method is described in this application embodiment.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] This application provides a target detection method, relating to the field of artificial intelligence technology. The target detection method provided in this application can be applied to scenarios including edge devices and servers. Edge devices, also known as edge computing devices, are the hardware that drives the application of edge computing in various industries. They are located close to the data source, enabling real-time data processing at very high speeds and performing various tasks, depending on the software applications or functions configured for them. Edge computing devices can be terminals such as smartphones, drones, tablets, laptops, and desktop computers, or edge servers deployed near the data generation point, or dedicated edge computing devices. The server can be configured as a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0073] Please see Figure 1 , Figure 1 This is a schematic diagram of the steps of a target detection method provided in an embodiment of this application. The target detection method is executed by an edge device, including but not limited to steps S101 to S107.

[0074] Step S101: Acquire video stream through camera unit.

[0075] It should be understood that the form of the camera unit here can be diverse. For example, the edge device is a terminal with a camera unit; the camera unit is set in the edge device and integrated with the edge device; or the camera unit is set in an external device and the external device communicates with the edge device. Those skilled in the art can select a suitable camera unit to meet the needs according to the actual situation, and this application does not limit it.

[0076] It should be understood that the acquisition methods are diverse and vary depending on the camera unit's configuration; this application does not impose any limitations on this.

[0077] In some possible embodiments of this application, the edge device obtains a complete video stream through its own camera and uploads the video stream to the server.

[0078] Step S102: Target detection is performed on each video frame in the video stream using a target detection model located on the edge device, so as to generate target bounding boxes in each video frame.

[0079] It should be understood that the target bounding boxes are recorded in coordinate form, and the visualization of the target bounding boxes is diverse. For example, when different objects are detected, all target bounding boxes are visualized. Those skilled in the art can choose the appropriate visualization of the target bounding boxes according to actual detection needs and model training needs, and this application does not limit it in this regard.

[0080] It should be understood that the types of target bounding box visualizations here are diverse. For example, real target bounding boxes and suspected pseudo-target bounding boxes may be visualized using bounding boxes of different colors. This application does not limit this.

[0081] It should be understood that the number of each type of target bounding box is varied, and there may be one or more. Those skilled in the art can set the number of each type of target bounding box acquired in the current video frame according to the actual situation, and this application does not limit this.

[0082] Step S103: Traverse each video frame in the video stream and perform the following processing on the currently traversed video frame: If there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frame, generate first detection bounding boxes that match the suspected false target bounding boxes in the first M video frames and the last N video frames of the currently traversed video frame, respectively. Calculate the correlation degree based on the first detection bounding boxes and the real target bounding boxes. If it is determined that the currently traversed video frame belongs to the hard case based on the correlation degree, add the currently traversed video frame to the hard case set.

[0083] It should be understood that, here, suspected false targets refer to targets with high pixel value similarity to real targets. For example, in a certain frame, a face and a hand appear at the same time. The face in that frame is the real target. Because the skin color is similar, the pixel value similarity is high. The hand may be mistaken for the face. In this case, the hand is a suspected false target before analysis.

[0084] It should be understood that M here is diverse. If the currently traversed video frame is the first frame, then M equals 0; if the currently traversed video frame is not the first frame, then M here is a preset value greater than 1. Those skilled in the art can set this preset value according to actual needs, and this application does not limit it.

[0085] It should be understood that N here is diverse. If the currently traversed video frame is the last frame, then N equals 0; if the currently traversed video frame is not the last frame, then N here is a preset value greater than 1. Those skilled in the art can set this preset value according to actual needs, and this application does not limit it in this regard.

[0086] It should be understood that the algorithms for calculating the degree of relevance here are diverse, such as the Structure Similarity Index Measure (SSIM) algorithm, or the intersection-union algorithm. Those skilled in the art can choose the appropriate algorithm to calculate the degree of relevance according to actual needs, and this application does not limit it.

[0087] It should be understood that determining whether the current traversed video frame belongs to a difficult case based on the degree of relevance here refers to comparing the degree of relevance with a preset threshold for judgment.

[0088] Step S104: Send the set of difficult examples to the server so that the server can train a copy of the target detection model based on the set of difficult examples and obtain model update data.

[0089] It should be understood that the number of video frames in the difficult example set here is a preset value. When the number of video frames reaches the preset value, the difficult example set is sent to the server. Those skilled in the art can set the preset value according to the actual situation, and this application does not limit it.

[0090] It should be understood that the copy here refers to a copy of the object detection model that has already performed object detection, not the original model without any weight parameters adjusted.

[0091] It should be understood that the model update data here refers to the updated values ​​of some parameters in the current object detection model.

[0092] It should be understood that the set of difficult examples sent to the server here includes not only the difficult example video frames, but also a video frame from the corresponding preceding and following frames in the difficult example. There are various ways to obtain the video frame from the corresponding preceding and following frames, such as random sampling; or sampling according to relevance. Those skilled in the art can obtain the video frame according to actual needs, and this application does not limit it in this way.

[0093] Step S105: Receive model update data sent by the server.

[0094] Step S106: Update the target detection model based on the model update data to obtain the updated target detection model.

[0095] In some embodiments, before updating the target detection model according to the model update data to obtain the updated target detection model, the method further includes: obtaining a target detection backup model; updating the target detection backup model according to the model update data to obtain the updated target detection backup model; and performing target detection on the video stream acquired by the camera unit using the updated target detection backup model.

[0096] Understandably, the edge device has two identical object detection models: a service model and a backup model. The service model continues to provide object detection services when data is updated. During an update, the service model continues to provide services, while the backup model updates itself based on the updated data. Once the update is complete, the updated backup model (the so-called updated object detection model) is put into service as the new service model, and the old service model is discontinued. Then, the old service model is updated again based on the updated data and becomes the new backup model.

[0097] Step S107: Perform target detection on the video stream acquired by the camera unit using the updated target detection model.

[0098] It should be understood that the updated object detection model here refers to the object detection model updated based on the backup model and updated data, not the object detection model updated based on the service model and updated data.

[0099] It should be understood that the video stream here is diverse. It can be the video stream in step S101, which is detected using the updated object detection model, or it can be different from step S101, where a new video stream is acquired using a camera unit that is different from the old video stream. Those skilled in the art can use the updated object detection model to perform object detection on a specific video stream according to the actual situation, and this application does not limit it in this regard.

[0100] Steps S101 to S107, as illustrated in this embodiment, involve detecting targets in the video frames of the acquired video stream, generating target bounding boxes, and then checking if the target bounding boxes exist in the preceding and following video frames. If not, a first detected bounding box is generated in the preceding and following frames, and its correlation with the real target bounding box is calculated. If the correlation indicates that the current video frame is a hard example, the current video frame is added to the hard example set. The hard example set is sent to the server to obtain model update data sent by the server. After obtaining the update data, the model is updated, and the updated model is used for video stream detection. By detecting unstable target bounding boxes, the detection error existing in the current model is obtained. The error is sent to the server to retrain the model to obtain update data. The updated data is used to update the current model, improving the model's ability to recognize targets and its robustness in recognizing targets in environments with similar objects.

[0101] Please see Figure 2 , Figure 2 for Figure 1 A schematic diagram of the sub-steps of step S103. In some possible embodiments of this application, step S103 includes, but is not limited to, the following sub-steps.

[0102] Step S201: Obtain the target bounding box that exists in the currently traversed video frame.

[0103] It should be understood that the target bounding boxes obtained here are not categorized into different types; they are not classified as real target bounding boxes or suspected pseudo-target bounding boxes.

[0104] It should be understood that the number of target bounding boxes here is diverse; that is, the target bounding boxes that exist in the current video frame are all the target bounding boxes.

[0105] Step S202: For each acquired target bounding box, determine whether the target bounding box exists in adjacent video frames.

[0106] It should be understood that adjacent video frames here refer to the M video frames before and the N video frames after the current video frame.

[0107] It should be understood that the use of whether or not a target exists in adjacent video frames is based on the instability of pseudo-targets. When a pseudo-target is identified, it is not like a real target. Real targets can be identified in every frame, but pseudo-targets are identified with a probability. An object that is a pseudo-target may be identified as a target in the current frame, but not in the previous or subsequent frame. Based on this, it can be determined that the current target is a pseudo-target.

[0108] Step S203: If the target bounding box exists in an adjacent video frame, then the target bounding box is determined as the real target bounding box.

[0109] Specifically, when a target bounding box exists in an adjacent video frame, the target in that bounding box appears stably in the adjacent video frame and is considered a real target. The bounding box that selects the real target is then determined as the real target bounding box.

[0110] Step S204: If the target bounding box does not exist in the adjacent video frame, then the target bounding box is determined as a suspected false target bounding box.

[0111] It should be understood that "not existing in adjacent video frames" here means that the target bounding box does not exist, not that the object in the target bounding box does not exist. The object in the target bounding box exists but is not detected as a target in adjacent video frames.

[0112] The following example illustrates this; please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the detection process provided in an embodiment of this application.

[0113] exist Figure 3 The video consists of three frames: the middle frame is the currently traversed frame, the left frame is the previous frame, and the right frame is the next frame. In this example, the hand is falsely detected, while the head is the target to be detected. In step S201, the bounding boxes of the identified targets are obtained, namely the bounding boxes of the head and the hand. Both the hand and the head appear in both the previous and next frames, and the bounding box for detecting the head appears in both frames, but the bounding box for detecting the hand does not appear. Therefore, the head bounding box is determined to be the true target bounding box, and the hand bounding box is determined to be the suspected target bounding box.

[0114] This application's embodiments determine the type of target bounding box by whether it exists in adjacent video frames, thereby improving the efficiency of hard example mining, reducing sample labeling time, and improving model training efficiency.

[0115] Please see Figure 4 , Figure 4 for Figure 1 A schematic diagram of another sub-step of step S103. In some possible embodiments of this application, after step S204, the generation of the first detection bounding box in step S103 includes, but is not limited to, the following sub-steps.

[0116] Step S301: For each of the first M video frames and the last N video frames, a template matching algorithm is used to find the region in the video frame that matches the bounding box of the suspected false target.

[0117] It should be understood that the template used in the template matching algorithm is the target in the suspected pseudo-target bounding box in step S204.

[0118] It should be understood that the search for regions that match the bounding boxes of suspected false targets here refers to searching for a target that is the same as or similar to the template in each of the first M video frames and the last N video frames.

[0119] Step S302: Determine the first detection bounding box based on the found region.

[0120] It should be understood that the generation method here is the same as the generation method of the target bounding box in step S102.

[0121] It should be understood that the number of regions to be searched here is varied and corresponds to their occurrence in the video frame. The number of first detection bounding boxes corresponds to the number of regions to be searched.

[0122] By using a template matching algorithm to find matching regions in consecutive video frames and determining the first detection bounding box based on the matching regions, the efficiency of finding suspected false targets is improved and the consumption of computing resources on edge devices is reduced.

[0123] Please see Figure 5 , Figure 5 for Figure 1 A schematic diagram of another sub-step of step S103. In some possible embodiments of this application, after step S302 and before calculating the relevance, step S103 may include, but is not limited to, the following steps.

[0124] Step S401: For each of the first M video frames and the last N video frames, enlarge the first detection bounding box and generate a second detection bounding box in the video frame.

[0125] It should be understood that there are various ways to enlarge the bounding box here. For example, the first detection bounding box can be detected according to a preset area method; another example is to enlarge the two sides of the first detection bounding box according to a preset ratio. Those skilled in the art can choose an appropriate enlargement method to enlarge the first detection bounding box according to actual needs. This application does not limit this.

[0126] It should be understood that the number of bounding boxes for enlarging the first detection bounding box and generating the second detection bounding box here corresponds to the number in step S302, where all the first detection bounding boxes in step S302 are enlarged and the second detection bounding boxes are generated respectively.

[0127] Continue using Figure 3 Provide examples

[0128] Taking the next frame as an example, the first detection bounding box is determined according to the template matching algorithm. Based on the first detection bounding box, the method is expanded to obtain the second detection bounding box. The previous frame is similar, and will not be elaborated here.

[0129] In some possible embodiments of this application, the boundary of the first detection bounding box is expanded by 100 pixels, and the new boundary is 100 pixels away from the old boundary.

[0130] Step S402: Calculate the normalized cross-correlation (NCC) value based on the second detected bounding box and the suspected false target bounding box. If the NCC value is greater than or equal to the first preset threshold, the video frame is used as the reserved video frame.

[0131] It should be understood that the value of the first preset threshold here is varied, and those skilled in the art can set a specific value according to actual needs. This application does not limit this value.

[0132] It should be understood that the first preset threshold here is used to indicate the degree to which the second detected bounding box is the same as the suspected false target bounding box. The calculated NCC value here is used to verify whether the template matching result in step S301 is reliable. If the NCC value is greater than or equal to the first preset threshold, it means that the target in the second detected bounding box with the calculated NCC value is the same target as the target in the suspected false target bounding box, that is, both are suspected false targets.

[0133] By using a normalized cross-correlation algorithm to calculate the similarity between the second detected bounding box and the suspected false target bounding box, useful frames from the preceding and following frames are selected for analysis, reducing the impact of template matching errors caused by large changes between frames on subsequent difficult case discovery.

[0134] Please see Figure 6 , Figure 6 for Figure 1 A schematic diagram of another sub-step of step S103. In some possible embodiments of this application, after step S402, the calculation of the relevance in step S103 includes, but is not limited to, the following steps.

[0135] Step S501: For the retained video frames, calculate the intersection-union ratio (IUU) between the first detected bounding box and the real target bounding box in the retained video frames.

[0136] It should be understood that the calculation of the intersection-union ratio here refers to the calculation of the intersection-union ratio of all the first detected bounding boxes and the ground target bounding boxes in the same video frame. The intersection-union ratio is calculated once for each first detected bounding box and the ground target bounding box.

[0137] Step S502: Determine the correlation between the first detected bounding box and the real target bounding box based on the intersection-union ratio.

[0138] It should be understood that the determination of the relevance here is based on a preset threshold. When the intersection-union ratio is less than the threshold, it means that the target in the first detection bounding box is not the same target as the real target bounding box, and the suspected false target in the first detection bounding box is determined as a false target. When the intersection-union ratio is greater than the threshold, it means that the target in the first detection bounding box is the same target as the real target bounding box, and the suspected false target in the first detection bounding box is determined as a real target.

[0139] By calculating the intersection-union ratio (IUGR), the degree of correlation between the first detection bounding box and the target in the real target bounding box is determined, thereby identifying difficult examples. This improves the efficiency and quality of acquiring difficult examples and reduces the systematic errors in model recognition caused by the similarity of targets in difficult examples during subsequent model training.

[0140] Please see Figure 7 , Figure 7 for Figure 1 A schematic diagram of the sub-steps of step S106. In some possible embodiments of this application, step S106 includes, but is not limited to, the following sub-steps.

[0141] Step S601: The teacher model is used to label the video frames in the difficult example set to obtain the labeled target bounding boxes corresponding to the video frames.

[0142] It should be understood that the teacher model here is different from the object detection model. The teacher model is only used to label video frames in the hard example set and add labeled object bounding boxes to each video frame.

[0143] It should be understood that the types of teacher models here are diverse, such as the DeeplabV3 model and the Xception65 model; those skilled in the art can choose a suitable model as the teacher model according to the actual situation, and this application does not limit it.

[0144] Step S602: Input the video frames in the hard example set into the copy of the object detection model to obtain the predicted object bounding box.

[0145] It should be understood that the predicted target bounding box of each video frame here is the same as the target bounding box generated in step S102.

[0146] Step S603: Determine the loss value based on the labeled target bounding box and the predicted target bounding box corresponding to each video frame in the difficult example set.

[0147] It should be understood that the loss value here refers to the loss value obtained by combining the loss values ​​corresponding to each video frame. That is, the loss value is determined by calculating the labeled target bounding box and the predicted target bounding box according to the preset loss function, statistically calculating the loss value, and determining the final loss value.

[0148] It should be understood that there are various ways to determine the final loss value. For example, the final loss value can be determined by the average of the various loss values. Those skilled in the art can choose an appropriate method for determining the loss value according to the actual situation, and this application does not limit it.

[0149] Step S604: Adjust the model parameters of the copy of the object detection model according to the loss value until the preset training termination condition is met.

[0150] It should be understood that the model parameters here are diverse, and can be coefficients in the loss function or weight parameters in each network layer of the model. This application does not limit them.

[0151] It should be understood that the preset training termination conditions are diverse, and those skilled in the art can set specific training termination conditions according to the actual situation. This application does not limit this.

[0152] The solution in this application embodiment can be applied to security scenarios in the financial industry. For example, a video stream captured by a bank surveillance camera unit is acquired, and target detection is performed on each video frame in the video stream using a target detection model located on the edge device to generate target bounding boxes in each video frame. Each video frame in the video stream is traversed, and the following processing is performed on the currently traversed video frame: if there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frame, a first detection bounding box matching the suspected false target bounding box is generated in the first M video frames and the last N video frames of the currently traversed video frame, respectively. The correlation is calculated based on the first detection bounding box and the real target bounding box. If the correlation is... Based on the correlation, the currently traversed video frame is determined to be a difficult example. The currently traversed video frame is added to the difficult example set, where M is greater than or equal to 0 and N is greater than or equal to 0. The difficult example set is sent to the server, so that the server trains a copy of the target detection model based on the difficult example set, obtaining model update data. The server receives the model update data. The target detection model on the edge device is updated based on the model update data, resulting in an updated target detection model. The edge device uses the updated target detection model to perform target detection on the video stream acquired by the bank's surveillance camera unit, thereby performing security monitoring of the bank environment.

[0153] This application embodiment detects targets in video frames of the acquired video stream, generates target bounding boxes, and then checks whether the target bounding boxes exist in the preceding and following video frames. If not, it generates first detected bounding boxes in the preceding and following frames and calculates their correlation with the real target bounding boxes. If the correlation indicates that the current video frame is a hard example, it is added to the hard example set. The hard example set is sent to the server to obtain model update data. After obtaining the update data, the model is updated, and the updated model is used for video stream detection. By detecting unstable target bounding boxes, the detection errors existing in the current model are obtained. The errors are sent to the server to retrain the model to obtain updated data. The updated data is used to update the current model, improving the model's ability to recognize targets and its robustness in recognizing targets in environments with similar objects, thereby improving the reliability of security monitoring in the financial industry.

[0154] Please see Figure 8 , Figure 8 This application also provides a schematic diagram of a target detection device, which can implement the above-described target detection method. The device 700 includes:

[0155] The video stream acquisition module 701 is used to acquire a video stream through the camera unit;

[0156] The first target detection module 702 is used to perform target detection on each video frame in the video stream using a target detection model located on the edge device, so as to generate target bounding boxes in each video frame.

[0157] The difficult example mining module 703 is used to traverse each video frame in the video stream and perform the following processing on the currently traversed video frame: if there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frame, generate first detection bounding boxes that match the suspected false target bounding boxes in the first M video frames and the last N video frames of the currently traversed video frame, respectively, calculate the correlation degree based on the first detection bounding boxes and the real target bounding boxes, and if it is determined that the currently traversed video frame belongs to a difficult example based on the correlation degree, add the currently traversed video frame to the difficult example set, where M is greater than or equal to 0 and N is greater than or equal to 0;

[0158] The sending module 704 is used to send the set of difficult examples to the server so that the server can train a copy of the target detection model based on the set of difficult examples and obtain model update data.

[0159] The receiving module 705 is used to receive model update data sent by the server;

[0160] Update module 706 is used to update the target detection model according to the model update data to obtain the updated target detection model;

[0161] The second target detection module 707 is used to perform target detection on the video stream acquired by the camera unit using the updated target detection model.

[0162] The specific implementation of this target detection device is basically the same as the specific embodiment of the target detection method described above, and will not be repeated here.

[0163] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the target detection method described above. This electronic device can be any intelligent edge device, including tablet computers, in-vehicle computers, etc.

[0164] Please see Figure 9 , Figure 9 This application provides a schematic diagram of the structure of an electronic device 800, which includes:

[0165] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0166] The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 according to an embodiment of this application for a target detection method.

[0167] The 803 input / output interface is used to implement information input and output.

[0168] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0169] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0170] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0171] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned target detection method.

[0172] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] This application provides a target detection method, a target detection device, an electronic device, and a storage medium. It detects targets in video frames of an acquired video stream, generates target bounding boxes, and then checks if the target bounding boxes exist in preceding and following video frames. If not, it generates first detected bounding boxes in the preceding and following frames and calculates their correlation with the real target bounding boxes. If the correlation indicates that the current video frame is a hard example, it adds the current video frame to a hard example set. The hard example set is sent to a server to obtain model update data. After obtaining the update data, the model is updated, and the updated model is used for video stream detection. By detecting unstable target bounding boxes, the detection error in the current model is obtained. The error is sent to the server to retrain the model to obtain updated data. The updated data is used to update the current model, improving the model's ability to recognize targets and its robustness in recognizing targets in environments with similar objects.

[0174] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0175] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0178] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0179] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0181] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0183] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A target detection method, characterized in that, Applied to edge devices, the method includes the following steps: Video streams are acquired through camera units; The target detection model located on the edge device is used to perform target detection on each video frame in the video stream to generate target bounding boxes in each video frame. Each video frame in the video stream is traversed, and the following processing is performed on the currently traversed video frame: Target bounding boxes are obtained in the currently traversed video frame. For each obtained target bounding box, it is determined whether the target bounding box exists in adjacent video frames. If the target bounding box exists in the adjacent video frames, it is determined as a real target bounding box. If the target bounding box does not exist in the adjacent video frames, it is determined as a suspected false target bounding box. If both real and suspected false target bounding boxes exist in the currently traversed video frame, a first detection bounding box matching the suspected false target bounding box is generated in the first M and last N video frames of the currently traversed video frame, respectively. If the intersection-union ratio (IUU) of the correlation between the first detection bounding box and the real target bounding box is less than a threshold, the currently traversed video frame is determined to be a hard case, and the currently traversed video frame is added to the hard case set, where M is greater than or equal to 0, and N is greater than or equal to 0. The set of difficult examples is sent to the server so that the server can train a copy of the object detection model based on the set of difficult examples and obtain model update data. Receive model update data sent by the server; The target detection model is updated based on the model update data to obtain the updated target detection model; The updated target detection model is used to perform target detection on the video stream acquired by the camera unit.

2. The target detection method according to claim 1, characterized in that, Before calculating the correlation between the first detected bounding box and the true target bounding box, the method further includes: For each of the first M video frames and the last N video frames, the first detection bounding box is enlarged, and a second detection bounding box is generated in the video frame; The normalized cross-correlation (NCC) value is calculated based on the second detected bounding box and the suspected false target bounding box. If the NCC value is greater than or equal to the first preset threshold, the video frame is selected as a reserved video frame.

3. The target detection method according to claim 1, characterized in that, The step of generating a first detection bounding box that matches the suspected false target bounding box in the first M video frames and the last N video frames of the currently traversed video frame includes: For each of the first M video frames and the last N video frames, a template matching algorithm is used to find the region in the video frame that matches the bounding box of the suspected false target. The first detection bounding box is determined based on the found region.

4. The target detection method according to claim 1, characterized in that, The step of training a copy of the object detection model based on the set of difficult examples to obtain model update data includes: The teacher model is used to label the video frames in the difficult example set to obtain the labeled target bounding boxes corresponding to the video frames; The video frames in the set of difficult examples are input into a copy of the target detection model to obtain the predicted target bounding box; The loss value is determined based on the labeled target bounding box and the predicted target bounding box corresponding to each video frame in the difficult example set; The model parameters of the copy of the target detection model are adjusted according to the loss value until the preset training termination condition is met.

5. The target detection method according to claim 1, characterized in that, Before updating the target detection model based on the model update data to obtain the updated target detection model, the method further includes: Obtain the target detection backup model; The target detection backup model is updated based on the model update data to obtain the updated target detection backup model. The updated target detection backup model is used to perform target detection on the video stream acquired by the camera unit.

6. A target detection device, characterized in that, The device includes: The video stream acquisition module is used to acquire video streams through the camera unit; The first target detection module is used to perform target detection on each video frame in the video stream using a target detection model located on the edge device, so as to generate target bounding boxes in each video frame. The difficult example mining module is used to traverse each video frame in the video stream and perform the following processing on the currently traversed video frame: obtain the target bounding boxes present in the currently traversed video frame; for each obtained target bounding box, determine whether the target bounding box exists in adjacent video frames; if the target bounding box exists in the adjacent video frames, then the target bounding box is determined as a real target bounding box; if the target bounding box does not exist in the adjacent video frames, then the target bounding box is determined as a suspected false target bounding box; if there are real target bounding boxes and suspected false target bounding boxes in the currently traversed video frame, generate a first detection bounding box matching the suspected false target bounding box in the first M video frames and the last N video frames of the currently traversed video frame, respectively; if the intersection-union ratio of the first detection bounding box and the real target bounding box is less than a threshold, then determine that the currently traversed video frame belongs to a difficult example and add the currently traversed video frame to the difficult example set, where M is greater than or equal to 0 and N is greater than or equal to 0; The sending module is used to send the set of difficult examples to the server so that the server can train a copy of the target detection model based on the set of difficult examples and obtain model update data. A receiving module is used to receive model update data sent by the server; The update module is used to update the target detection model according to the model update data to obtain the updated target detection model; The second target detection module is used to perform target detection on the video stream acquired by the camera unit using the updated target detection model.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing computer programs or instructions, and the processor executing the computer programs or instructions to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 5.

Images