Video-based unsupervised hard-example data mining method, device, medium and equipment

Abstract

The present invention discloses a video-based unsupervised difficult example data mining method, comprising: using a first detection model to be optimized to perform frame-by-frame detection on an unlabeled video to generate a first detection result; according to the first detection result, selecting two adjacent frames of images that are not continuous to form a difficult example image pair; using a second detection model to detect the first frame of the difficult example image pair to obtain a second detection result; and judging the type of difficult examples in the difficult example image pair according to the second detection result. The present invention selects useful images in a targeted manner and avoids the generation of a large number of repeated simple images.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a video-based unsupervised hard-example data mining method, device, medium and equipment. Background Art

[0002] Deep learning training requires a large amount of data. One method is to evenly cut frames in the video and then send them to manual labeling. However, this method is not targeted and will dig out many useless pictures (for example, the neural network effect is already very good in some scenes). Sending these useless pictures for manual labeling not only causes a huge waste of human resources, but also makes some very valuable pictures drown in useless pictures, which ultimately leads to no improvement in model performance. Another method is to run the model to be optimized in the original video for testing, manually observe which scenes are not effective, and then select the area in the video. This method is targeted when selecting pictures, but the labor cost is too high. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a video-based unsupervised hard-example data mining method, device, medium and equipment to solve the problems existing in the prior art.

[0004] To achieve the above objectives and other related objectives, the present invention provides a video-based unsupervised hard example data mining method, comprising:

[0005] Using the first detection model to be optimized to perform frame-by-frame detection on the unlabeled video to generate a first detection result;

[0006] According to the first detection result, two adjacent frames of images that are not continuous are selected to form a difficult image pair;

[0007] Using the second detection model to detect the first frame of the difficult image pair, to obtain a second detection result;

[0008] The type of difficult examples present in the difficult example image pair is determined according to the second detection result.

[0009] Optionally, the difficult example types include difficult positive examples and difficult negative examples.

[0010] Optionally, the method for determining two adjacent frames of images that are not continuous includes:

[0011] Get the first intersection-over-union ratio of two adjacent frames of images;

[0012] Determining whether two adjacent frames of images are continuous according to the first intersection-to-union ratio;

[0013] If the first IoU ratio is less than a first IoU ratio threshold, then two adjacent frames of images are not continuous, otherwise they are continuous.

[0014] Optionally, the difficult-example image pair includes a first frame image with a first detection box and a second frame image without the first detection box.

[0015] Optionally, if the second detection result includes a second detection box, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second detection result does not include a second detection box, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

[0016] Optionally, if the second detection result includes a second detection frame, the method further includes:

[0017] Calculate a second intersection-over-union ratio of the first detection frame and the second detection frame;

[0018] The type of difficult examples in the difficult example image pair is determined according to the second intersection-over-union ratio.

[0019] Optionally, if the second intersection-and-union ratio is greater than or equal to a second intersection-and-union ratio threshold, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second intersection-and-union ratio is less than the second intersection-and-union ratio threshold, there is a difficult positive example in the first frame of the picture, and the first frame of the picture is retained.

[0020] Optionally, the first detection model is retrained using a data set consisting of retained difficult positive example images and difficult negative example images to optimize the first detection model.

[0021] Optionally, when retraining the first detection model, the second detection box area in the difficult positive example image and / or the first detection box area in the difficult negative example image and / or the intersection area of ​​the first detection box area and the second detection box area in the difficult negative example image are retrained.

[0022] To achieve the above-mentioned and other related purposes, the present invention provides a video-based unsupervised hard example data mining device, comprising:

[0023] A first detection module, used to detect the unlabeled video frame by frame using the first detection model to be optimized to generate a first detection result;

[0024] a difficult-example picture pair generation module, configured to select two adjacent frames of pictures that are not continuous to form a difficult-example picture pair according to the first detection result; wherein the first frame of the two adjacent frames of pictures includes the first detection frame, and the second frame of the pictures does not include the first detection frame;

[0025] A second detection module, configured to detect the first frame of the difficult image pair using a second detection model to obtain a second detection result;

[0026] A difficult example identification module is used to determine the type of difficult examples in the difficult example image pair according to the second detection result.

[0027] Optionally, the difficult example types include difficult positive examples and difficult negative examples.

[0028] Optionally, the difficult example image pair generation module includes:

[0029] A first intersection-and-union ratio calculation submodule is used to obtain a first intersection-and-union ratio of two adjacent frames of images when the second detection result contains a second detection frame;

[0030] The continuity judgment submodule is used to judge whether two adjacent frames of pictures have continuity according to the first intersection-and-union ratio; if the first intersection-and-union ratio is less than a first intersection-and-union ratio threshold, the two adjacent frames of pictures do not have continuity, otherwise they have continuity.

[0031] Optionally, the difficult-example image pair includes a first frame image with a first detection box and a second frame image without the first detection box.

[0032] Optionally, if the second detection result includes a second detection box, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second detection result does not include a second detection box, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

[0033] Optionally, the difficult example image pair generation module further includes:

[0034] A second IoU calculation module, configured to calculate a second IoU ratio between the first detection frame and the second detection frame;

[0035] The difficult example determination module determines the type of difficult examples in the difficult example image pair according to the second IOR.

[0036] Optionally, if the second intersection-over-union ratio is greater than or equal to a second intersection-over-union ratio threshold, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second intersection-over-union ratio is less than the second intersection-over-union ratio threshold, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

[0037] To achieve the above-mentioned purpose and other related purposes, the present invention also provides a video-based unsupervised hard example data mining device, comprising:

[0038] one or more processors; and

[0039] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the device to perform one or more of the methods described above.

[0040] To achieve the above objectives and other related objectives, the present invention also provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the device to execute one or more of the aforementioned methods.

[0041] As described above, the present invention provides a video-based unsupervised hard-example data mining method, device, medium and equipment, which have the following beneficial effects:

[0042] The present invention provides a video-based unsupervised difficult example data mining method, comprising: using a first detection model to be optimized to perform frame-by-frame detection on an unlabeled video to generate a first detection result; according to the first detection result, selecting two adjacent frames of images that are not continuous to form a difficult example image pair; using a second detection model to detect the first frame of the difficult example image pair to obtain a second detection result; and judging the type of difficult examples in the difficult example image pair according to the second detection result. The present invention selects useful images in a targeted manner and avoids the generation of a large number of repeated simple images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a video-based unsupervised hard-example data mining method according to an embodiment of the present invention;

[0044] Figure 2 A flowchart of a method for determining two adjacent frames of images that are not continuous according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of calculating the intersection-over-union (IOU) ratio according to an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the structure of a video-based unsupervised hard-example data mining device according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of a difficult example image pair generation module according to an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the hardware structure of a terminal device in an embodiment of the present invention;

[0049] Figure 7 The figure is a schematic diagram of the hardware structure of a terminal device in one embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0052] Videos are generally continuous in time (adjacent frames) and space, and it is impossible for an object to suddenly disappear or appear. An object frame that suddenly appears in a continuous video frame is likely to be a false positive. Taking the face detection model as an example, if the model's detection results (face frames) are continuous in time and space, then the video is a simple video for the model and has no great significance. On the contrary, if the model's detection results are not continuous in time and space, it means that there are useful images (difficult positive examples or difficult negative examples) for the model. Difficult positive examples are considered to be detected but are not detected, and difficult negative examples are considered to be detected but should not be detected.

[0053] Therefore, the present invention proposes a method of using the existing model to detect on adjacent video frames, to mine useful images in the video in a targeted manner and to train. This method solves the problem that evenly cutting frames in the video cannot mine useful images in the video in a targeted manner, saves the cost of manual annotation and better improves the performance of the model.

[0054] like Figure 1 As shown, a video-based unsupervised hard example data mining method includes:

[0055] S11 uses the first detection model to be optimized to perform frame-by-frame detection on the unlabeled video to generate a first detection result;

[0056] S12, according to the first detection result, selecting two adjacent frames of images that are not continuous to form a difficult example image pair;

[0057] S13 uses the second detection model to detect the first frame of the difficult image pair to obtain a second detection result;

[0058] S14 determines the type of difficult examples in the difficult example image pair according to the second detection result.

[0059] The present invention selects useful pictures in a targeted manner, thus avoiding the generation of a large number of repeated simple pictures.

[0060] In step S14, the difficult example types include difficult positive examples and difficult negative examples; a difficult positive example is an area with a target but no target is detected, and a difficult negative example is a false detection area including an area with no target but a target is detected or an incorrect target is detected.

[0061] In step S11, the first detection model is a model to be optimized generated by training on an existing data set, and its performance is not optimal. The first detection model can be considered as a detector. The first detection model includes but is not limited to Faster RCNN (faster image region-based convolutional neural network), SSD (Single Shot MultiBoxDetector, a target detection model), YOLO (You LookOnce, a target detection model) and RetinaNet, etc. Existing data sets can be used, such as the Caltech pedestrian detection data set or the WIDER FACE face data set.

[0062] In one embodiment, if Figure 2 As shown, the method for determining two adjacent frames of pictures that are not continuous includes:

[0063] S21 obtains a first intersection-over-union ratio of two adjacent frames of images;

[0064] S22 determines whether two adjacent frames of pictures have continuity according to the first IoU ratio; if the first IoU ratio is less than a first IoU ratio threshold, the two adjacent frames of pictures do not have continuity, otherwise they have continuity.

[0065] When the first detection model is used to detect the unlabeled video frame by frame, the video can be a video with a face downloaded from the Internet. When the video is detected, there is a detection result for each frame, which may include a detection frame or may not include a detection frame. The detection frame is defined as the first detection frame.

[0066] On the premise that two adjacent frames of images are not continuous, the two adjacent frames of images are used to form a difficult image pair, where the two frames of images meet the following conditions: the first frame of the image contains the first detection frame, and the second frame of the image does not contain the first detection frame.

[0067] In step S13, the second detection model is a pre-trained detection model, such as a trained neural network model for target detection; the detection model structure is relatively complex, but has the characteristics of high detection accuracy, and the detection result of the second detection model can be considered correct. The first frame of the difficult example picture pair is detected by the second detection model. If the detection result of the first frame includes the second detection frame, it can be considered that there is a target in the area of ​​the second detection frame itself; in the difficult example picture pair consisting of two adjacent frames, the second frame does not have the first detection frame, that is, the target should be detected here but the first detection model does not actually detect the target, therefore, it is considered that there is a difficult positive example in the area of ​​the second frame, and the second frame is retained at this time. If the detection result of the first frame does not include the second detection frame, it can be considered that there is no target in the area of ​​the second detection frame itself, and in the difficult example picture pair consisting of two adjacent frames, the first frame has the first detection frame, that is, the target should not be detected here but the first detection model actually detects the target, therefore, there is a difficult negative example in the area of ​​the first frame, and the first frame is retained at this time.

[0068] In one embodiment, if the second detection result includes a second detection frame, the method further includes:

[0069] Calculate a second intersection-over-union ratio of the first detection frame and the second detection frame; and determine the type of difficult examples in the difficult example image pair according to the second intersection-over-union ratio.

[0070] Among them, the intersection-over-union ratio is expressed by IOU, such as Figure 3 As shown, the intersection-over-union ratio represents the ratio of the intersection and union of the "first detection box" and the "second detection box", that is, IOU = S3 / (S1+S2).

[0071] The first frame of the difficult picture pair is detected by the second detection model. If the detection result of the first frame includes the second detection frame, it can be considered that there is a target in the area of ​​the second detection frame itself. In the difficult picture pair consisting of two adjacent frames, the first frame has the first detection frame. If the intersection and union ratio of the first detection frame and the second detection frame is greater than or equal to the set threshold, such as 0.2, it is considered that the detection result of the first detection model is relatively correct, and the target can be detected here. However, in the difficult picture pair consisting of two adjacent frames, the second frame does not have the first detection frame, that is, the target should be detected here, but the first detection model does not actually detect the target. Therefore, it is considered that there is a difficult positive example in the area of ​​the second frame, and the second frame is retained. If the intersection and union ratio of the first detection frame and the second detection frame is less than the set threshold, such as 0.2, it can be considered that although the first detection model detects the target, it only contains part of the target, and it can be considered that the detection of the target is wrong. Therefore, there is a difficult negative example in the area of ​​the first frame, and the first frame is retained.

[0072] In one embodiment, the first detection model is retrained using a data set consisting of retained hard positive example images and hard negative example images to optimize the first detection model.

[0073] When retraining the first detection model, the recognition ability of the second detection frame area in the difficult positive example picture or / and the first detection frame area in the difficult negative example picture or / and the intersection area of ​​the first detection frame area and the second detection frame area in the difficult negative example picture is enhanced.

[0074] like Figure 4 As shown, a video-based unsupervised hard example data mining device comprises:

[0075] A first detection module 41 is used to detect the unlabeled video frame by frame using the first detection model to be optimized to generate a first detection result;

[0076] A difficult picture pair generating module 42, configured to select two adjacent frames of pictures that are not continuous to form a difficult picture pair according to the first detection result;

[0077] A second detection module 43 is used to detect the first frame of the difficult image pair using a second detection model to obtain a second detection result;

[0078] The difficult example determination module 44 is used to determine the type of difficult examples in the difficult example image pair according to the second detection result.

[0079] The present invention selects useful pictures in a targeted manner, thus avoiding the generation of a large number of repeated simple pictures.

[0080] The types of difficult examples include difficult positive examples and difficult negative examples; difficult positive examples are areas that are not detected, and difficult negative examples are areas that are falsely detected.

[0081] The first detection model is generated by training on an existing data set, and the first detection model can be considered as a detector. The first detection model includes but is not limited to Faster RCNN (faster image region-based convolutional neural network), SSD (Single Shot MultiBox Detector, a target detection model), YOLO (You LookOnce, a target detection model) and RetinaNet, etc. Existing data sets can be used, such as the Caltech pedestrian detection data set or the WIDER FACE face data set.

[0082] When the first detection model is used to detect the unlabeled video frame by frame, the video can be a video with a face downloaded from the Internet. When the video is detected, there is a detection result for each frame, which may include a detection frame or may not include a detection frame. The detection frame is defined as the first detection frame.

[0083] In one embodiment, if Figure 5 As shown, the difficult example picture pair generation module includes:

[0084] A first IoU calculation submodule 51 is used to obtain a first IoU ratio of two adjacent frames of images;

[0085] The continuity judgment submodule 52 is used to judge whether two adjacent frames of pictures are continuous according to the first I / O ratio; if the first I / O ratio is less than a first I / O ratio threshold, the two adjacent frames of pictures are not continuous, otherwise they are continuous.

[0086] Among them, the intersection-over-union ratio is expressed by IOU, such as Figure 3 As shown, the intersection-over-union ratio represents the ratio of the intersection and union of the "first detection box" and the "second detection box", that is, IOU = S3 / (S1+S2).

[0087] On the premise that two adjacent frames of images are not continuous, the two adjacent frames of images are used to form a difficult image pair, where the two frames of images meet the following conditions: the first frame of the image contains the first detection frame, and the second frame of the image does not contain the first detection frame.

[0088] The second detection model is a pre-trained detection model, such as a trained neural network model for target detection; the detection model structure is relatively complex, but has the characteristics of high detection accuracy, and the detection result of the second detection model can be considered correct. The first frame of the difficult example image pair is detected using the second detection model. If the detection result of the first frame includes the second detection frame, it can be considered that there is a target in the area of ​​the second detection frame itself; in the difficult example image pair consisting of two adjacent frames, the second frame does not have the first detection frame, that is, the target should be detected here but the first detection model does not actually detect the target, therefore, it is considered that there is a difficult positive example in the area of ​​the second frame, and the second frame is retained at this time. If the detection result of the first frame does not include the second detection frame, it can be considered that there is no target in the area of ​​the second detection frame itself, and in the difficult example image pair consisting of two adjacent frames, the first frame has the first detection frame, that is, the target should not be detected here but the first detection model actually detects the target, therefore, there is a difficult negative example in the area of ​​the first frame, and the first frame is retained at this time.

[0089] In one embodiment, the difficult example image pair generation module includes:

[0090] A second intersection-over-union ratio calculation module, used for calculating a second intersection-over-union ratio between the first detection frame and the second detection frame when the second detection result contains a second detection frame;

[0091] The difficult example determination module determines the type of difficult examples in the difficult example image pair according to the second IOR.

[0092] Among them, the intersection-over-union ratio is expressed by IOU, such as Figure 3 As shown, the intersection-over-union ratio represents the ratio of the intersection and union of the "first detection box" and the "second detection box", that is, IOU = S3 / (S1+S2).

[0093] The first frame of the difficult picture pair is detected by the second detection model. If the detection result of the first frame includes the second detection frame, it can be considered that there is a target in the area of ​​the second detection frame itself. In the difficult picture pair consisting of two adjacent frames, the first frame has the first detection frame. If the intersection and union ratio of the first detection frame and the second detection frame is greater than or equal to the set threshold, such as 0.2, it is considered that the detection result of the first detection model is relatively correct, and the target can be detected here. However, in the difficult picture pair consisting of two adjacent frames, the second frame does not have the first detection frame, that is, the target should be detected here, but the first detection model does not actually detect the target. Therefore, it is considered that there is a difficult positive example in the area of ​​the second frame, and the second frame is retained. If the intersection and union ratio of the first detection frame and the second detection frame is less than the set threshold, such as 0.2, it can be considered that although the first detection model detects the target, it only contains part of the target, and it can be considered that the detection of the target is wrong. Therefore, there is a difficult negative example in the area of ​​the first frame, and the first frame is retained.

[0094] In one embodiment, the first detection model is retrained using a data set consisting of retained hard positive example images and hard negative example images to optimize the first detection model.

[0095] When retraining the first detection model, the recognition ability of the second detection frame area in the difficult positive example picture or / and the first detection frame area in the difficult negative example picture or / and the intersection area of ​​the first detection frame area and the second detection frame area in the difficult negative example picture is enhanced.

[0096] The embodiment of the present application also provides a device, which may include: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the device to execute Figure 1 In practical applications, the device can be used as a terminal device or a server. Examples of terminal devices may include: smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc. The embodiments of this application do not limit specific devices.

[0097] The present application also provides a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can execute the embodiment of the present application. Figure 1 Instructions for the steps in a method.

[0098] Figure 6 A schematic diagram of the hardware structure of a terminal device provided for an embodiment of the present application. As shown in the figure, the terminal device may include: an input device 1100, a first processor 1101, an output device 1102, a first memory 1103 and at least one communication bus 1104. The communication bus 1104 is used to realize the communication connection between the components. The first memory 1103 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage. Various programs can be stored in the first memory 1103 to complete various processing functions and implement the method steps of this embodiment.

[0099] Optionally, the first processor 1101 may be implemented, for example, as a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and the first processor 1101 is coupled to the input device 1100 and the output device 1102 via a wired or wireless connection.

[0100] Optionally, the input device 1100 may include multiple input devices, for example, at least one of a user interface for users, a device interface for devices, a software programmable interface, a camera, and a sensor. Optionally, the device interface for devices may be a wired interface for data transmission between devices, or a hardware insertion interface for data transmission between devices (such as a USB interface, a serial port, etc.); Optionally, the user interface for users may be, for example, a control button for users, a voice input device for receiving voice input, and a touch sensing device for users to receive user touch input (such as a touch screen with a touch sensing function, a touchpad, etc.); Optionally, the software programmable interface may be, for example, an entry for users to edit or modify programs, such as an input pin interface or an input interface of a chip; the output device 1102 may include output devices such as a display and a speaker.

[0101] In this embodiment, the processor of the terminal device includes functions for executing each module in each device. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0102] Figure 7 A schematic diagram of the hardware structure of a terminal device provided for one embodiment of the present application. Figure 7 Yes Figure 6 A specific embodiment in the implementation process. As shown in the figure, the terminal device of this embodiment may include a second processor 1201 and a second memory 1202.

[0103] The second processor 1201 executes the computer program code stored in the second memory 1202 to implement the above embodiment. Figure 1 The method.

[0104] The second memory 1202 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method used to operate on the terminal device, such as messages, pictures, videos, etc. The second memory 1202 may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage.

[0105] Optionally, the second processor 1201 is arranged in the processing component 1200. The terminal device may further include: a communication component 1203, a power component 1204, a multimedia component 1205, a voice component 1206, an input / output interface 1207 and / or a sensor component 1208. The specific components included in the terminal device are set according to actual needs, and this embodiment does not limit this.

[0106] The processing component 1200 generally controls the overall operation of the terminal device. The processing component 1200 may include one or more second processors 1201 to execute instructions to complete all or part of the steps in the above-mentioned data processing method. In addition, the processing component 1200 may include one or more modules to facilitate the interaction between the processing component 1200 and other components. For example, the processing component 1200 may include a multimedia module to facilitate the interaction between the multimedia component 1205 and the processing component 1200.

[0107] The power supply component 1204 provides power to various components of the terminal device. The power supply component 1204 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0108] The multimedia component 1205 includes a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0109] The voice component 1206 is configured to output and / or input voice signals. For example, the voice component 1206 includes a microphone (MIC), and when the terminal device is in an operating mode, such as a voice recognition mode, the microphone is configured to receive an external voice signal. The received voice signal can be further stored in the second memory 1202 or sent via the communication component 1203. In some embodiments, the voice component 1206 also includes a speaker for outputting voice signals.

[0110] The input / output interface 1207 provides an interface between the processing component 1200 and the peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include but are not limited to: volume buttons, start buttons, and lock buttons.

[0111] The sensor component 1208 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor component 1208 can detect the open / closed state of the terminal device, the relative positioning of the components, and the presence or absence of contact between the user and the terminal device. The sensor component 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, the sensor component 1208 may also include a camera, etc.

[0112] The communication component 1203 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, so that the terminal device can log in to a GPRS network and establish communication with a server via the Internet.

[0113] From the above, we can see that Figure 7 The communication component 1203, the voice component 1206, the input / output interface 1207, and the sensor component 1208 involved in the embodiment can all be used as Figure 6 Implementation method of the input device in the embodiment.

[0114] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A video-based unsupervised hard example data mining method, characterized in that: include: Using the first detection model to be optimized to perform frame-by-frame detection on the unlabeled video to generate a first detection result; According to the first detection result, two adjacent frames of images that are not continuous are selected to form a difficult image pair; Using the second detection model to detect the first frame of the difficult image pair, to obtain a second detection result; Determining the type of difficult examples in the difficult example image pair according to the second detection result; The method for determining two adjacent frames of pictures that are not continuous includes: Get the first intersection-over-union ratio of two adjacent frames of images; Determining whether two adjacent frames of images are continuous according to the first intersection-to-union ratio; If the first intersection-over-union ratio is less than a first intersection-over-union ratio threshold, the two adjacent frames of images are not continuous, otherwise they are continuous; The types of difficult examples include difficult positive examples and difficult negative examples; The difficult example picture pair includes a first frame picture with a first detection frame and a second frame picture without the first detection frame; If the second detection result includes the second detection box, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second detection result does not include the second detection box, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

2. The video-based unsupervised hard example data mining method according to claim 1, characterized in that: If the second detection result includes a second detection frame, the method further includes: Calculate a second intersection-over-union ratio of the first detection frame and the second detection frame; The type of difficult examples in the difficult example image pair is determined according to the second intersection-over-union ratio.

3. The video-based unsupervised hard example data mining method according to claim 2, characterized in that: If the second intersection-over-union ratio is greater than or equal to the second intersection-over-union ratio threshold, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second intersection-over-union ratio is less than the second intersection-over-union ratio threshold, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

4. The video-based unsupervised hard example data mining method according to claim 1 or 3, characterized in that: The first detection model is retrained using a data set consisting of the retained difficult positive example images and the difficult negative example images to optimize the first detection model.

5. The video-based unsupervised hard example data mining method according to claim 4, characterized in that: When retraining the first detection model, weighted processing is performed on the second detection frame area in the difficult positive example picture and / or the first detection frame area in the difficult negative example picture and / or the intersection area of ​​the first detection frame area and the second detection frame area in the difficult negative example picture.

6. A video-based unsupervised hard example data mining device, characterized in that: include: A first detection module, used to detect the unlabeled video frame by frame using the first detection model to be optimized to generate a first detection result; A difficult picture pair generation module, configured to select two adjacent frames of pictures that are not continuous to form a difficult picture pair according to the first detection result; A second detection module, configured to detect the first frame of the difficult image pair using a second detection model to obtain a second detection result; a difficult example identification module, configured to determine the type of difficult examples in the difficult example image pair according to the second detection result; The difficult example image pair generation module includes: A first intersection-and-union ratio calculation submodule is used to obtain a first intersection-and-union ratio of two adjacent frames of images when the second detection result contains a second detection frame; a continuity judgment submodule, used for judging whether two adjacent frames of pictures have continuity according to the first intersection-and-union ratio; if the first intersection-and-union ratio is less than a first intersection-and-union ratio threshold, the two adjacent frames of pictures do not have continuity, otherwise they have continuity; The types of difficult examples include difficult positive examples and difficult negative examples; The difficult example picture pair includes a first frame picture with a first detection frame and a second frame picture without the first detection frame; If the second detection result includes the second detection box, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second detection result does not include the second detection box, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

7. The video-based unsupervised hard example data mining device according to claim 6, characterized in that: The difficult example image pair generation module also includes: A second IoU calculation module, configured to calculate a second IoU ratio between the first detection frame and the second detection frame; The difficult example determination module determines the type of difficult examples in the difficult example image pair according to the second IOR.

8. The video-based unsupervised hard example data mining device according to claim 7, characterized in that: If the second intersection-over-union ratio is greater than or equal to the second intersection-over-union ratio threshold, there is a difficult positive example in the second frame of the picture, and the second frame of the picture is retained; if the second intersection-over-union ratio is less than the second intersection-over-union ratio threshold, there is a difficult negative example in the first frame of the picture, and the first frame of the picture is retained.

9. A video-based unsupervised hard example data mining device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the device to perform the method according to any one of claims 1-5.

10. One or more machine-readable media, characterized in that Instructions are stored thereon, which, when executed by one or more processors, cause the device to perform the method according to any one of claims 1 to 5.

Images