Object detection method and electronic device

Abstract

The present invention provides an object detection method and an electronic device, wherein the object detection method comprises: using an image processing circuit to determine whether an action occurs in an image to generate a determination result; based on the determination result, selectively using a specific frame to identify a target object to generate a recognition result, wherein the specific frame represents the position of the target object in a previous image; and based on the recognition result, selectively updating the information of the specific frame.

Description

Technical Field

[0001] The present invention relates to object detection, and in particular to an object detection method and an electronic device. Background Art

[0002] In recent years, object detection has become a widely discussed topic, and its purpose is to detect the location of a specific object in an input image. Currently, many object detection methods require scanning the entire input image (for example, using a sliding window with an image pyramid), which is quite computationally expensive. However, when the object to be detected has the characteristic of moving (such as a person or a car), object recognition can be performed only on the position with motion in the input image to greatly reduce the overall computing time and cost.

[0003] However, when the object to be detected stops moving, the movement disappears, so even if the object to be detected appears in the picture, it cannot be detected. In addition, when the object to be detected stops moving, the movement area may gradually shrink, resulting in incomplete information being captured and easily leading to misjudgment. Therefore, a novel method and electronic device are needed to improve the performance of object detection without side effects or with less side effects. Summary of the invention

[0004] The object of the present invention is to provide an object detection method (especially an object detection method applicable to static scenes) and an electronic device to improve the performance of object detection based on the detection of motion information.

[0005] At least one embodiment of the present invention provides an object detection method. The object detection method includes: using an image processing circuit to determine whether an action occurs in an image to generate a determination result; based on the determination result, selectively using a specific bounding box to identify a target object to generate a recognition result, wherein the specific bounding box represents the position of the target object in a previous image; and based on the recognition result, selectively updating information of the specific bounding box.

[0006] At least one embodiment of the present invention provides an electronic device. The electronic device includes a camera module, a storage device, and an image processing circuit coupled to the camera module and the storage device. The camera module can be used to acquire an image, and the storage device can be used to store information of a specific frame and a background model, wherein the specific frame represents the position of a target object in a previous image. In addition, the image processing circuit can be used to execute program code to perform object detection on the image, for example: the image processing circuit determines whether an action occurs in the image to generate a judgment result; based on the judgment result, the image processing circuit selectively uses the specific frame to identify the target object to generate a recognition result; and based on the recognition result, the image processing circuit selectively updates the information of the specific frame.

[0007] The object detection method and electronic device provided by the present invention provide corresponding frame adjustment mechanisms for situations with and without motion, respectively, to ensure that in various situations (such as when the object is moving, when the object is gradually still, when the object is completely still, etc.), it is possible to accurately determine whether the target object appears in the image. Compared with the related art, the embodiments of the present invention do not significantly increase the additional cost, so the present invention can improve the performance of object detection without or with less side effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0009] Figure 2 The following is a workflow of an object detection method according to an embodiment of the present invention.

[0010] Figure 3 FIG. 4 is a schematic diagram of candidate bounding boxes in an image according to an embodiment of the present invention.

[0011] Figure 4 FIG. 4 is a schematic diagram of candidate bounding boxes in an image according to an embodiment of the present invention.

[0012] Figure 5 FIG. 4 is a schematic diagram of candidate bounding boxes in an image according to an embodiment of the present invention.

[0013] Figure 6 FIG. 4 is a schematic diagram of candidate bounding boxes in an image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] Figure 11 is a schematic diagram of an electronic device 10 according to an embodiment of the present invention, wherein the electronic device 10 may include a camera module 120, a storage device 140, and an image processing circuit 160 coupled to the camera module 120 and the storage device 140. In the present embodiment, the camera module 120 may be used to acquire an image IMG, and the storage device 140 may be used to store information of a specific frame and a background model, wherein the specific frame represents the position of a target object (e.g., a person or a car) in a previous image. In addition, the image processing circuit 160 may be used to execute a program code 160C to perform object detection on the image IMG (e.g., detecting whether the target object appears in the image IMG). Specifically, the image processing circuit 160 may determine whether an action occurs in the image IMG to generate a determination result, for example, the image processing circuit 160 may determine an area in the image IMG where an object moves by using a frame difference between previous and subsequent frames or by establishing a background model. Based on the judgment result, the image processing circuit 160 can selectively use the specific border to identify the target object to generate a recognition result. For example, the image processing circuit 160 can identify the target object in the region through a type of neural network model established in the program code 160C, and output a corresponding confidence value, wherein the confidence value is positively correlated with the probability that the object in the region is the target object (for example, the larger the confidence value, the higher the probability that the object in the region is the target object). When the confidence value is greater than the confidence threshold, the image processing circuit 160 determines that the target object appears in the image IMG (especially the region). Based on the recognition result, the image processing circuit 160 can selectively update the information of the specific border (for example, the size and / or position of the specific border) to ensure that the latest information of the specific border can be used to identify the target object in the next image.

[0015] Figure 2 FIG. 1 is a workflow of an object detection method according to an embodiment of the present invention, wherein the object detection method is applicable to Figure 1 If the overall workflow can achieve the same or similar results, the electronic device 10 shown in FIG. Figure 2 The execution order of one or more steps in the workflow shown may be changed, or one or more steps in the workflow may be added, deleted or modified.

[0016] In step 210 , the image processing circuit 160 may obtain the image IMG through the camera module 120 .

[0017] In step 220 , the image processing circuit 160 may determine whether an action occurs in the image 300 to generate a determination result. If so, the process proceeds to step 231 ; otherwise, the process proceeds to step 241 .

[0018] In step 231, the image processing circuit 160 may obtain at least one candidate frame according to the position of the action in the image IMG. For example, when the entire target object is moving, the image processing circuit 160 may obtain a corresponding candidate frame (e.g., Figure 3 The border BB in the image 300 shown 0 For another example, when the target object (such as a person or a car) stops moving, only a partial area of ​​the target object may have movement, so the image processing circuit 160 can generate one or more corresponding candidate frames (such as Figure 4 The border BB in the image 400 is shown 1 BB 2 and BB 3 ). For ease of understanding, the following descriptions are based on Figure 3 The border BB shown 0 As an example of the specific bounding box, in particular, as an example of the specific bounding box with a sufficiently high confidence value (for example, higher than the confidence threshold).

[0019] In step 232, the image processing circuit 160 may use the at least one candidate frame to identify the target object, for example, determining whether the at least one candidate frame (eg, frame BB 1 BB 2 and BB 3 ) is the target object. If so, the process proceeds to step 233; otherwise, the process proceeds to step 234.

[0020] In step 233, the image processing circuit 160 may determine the size of the at least one candidate frame (eg, frame BB 1 BB 2 and BB 3 is greater than the size of the specific border (for example, border BB 0 ), or the size of at least one candidate border (e.g., border BB 1 BB 2 and BB 3 ) relative to that particular border (e.g. border BB 0 ) is lower than the overlap rate threshold. If so, the process proceeds to step 235; otherwise (for example, when the size of the at least one candidate frame is smaller than the size of the specific frame and the overlap rate of the at least one candidate frame relative to the specific frame is greater than the overlap rate threshold), the process proceeds to step 234. The overlap rate (IoU) is calculated as follows:

[0021] IoU=(A(BBOXcurrent ∩BBOX previous )) / A(BBOX previous );

[0022] The symbol A() represents the area, BBOX current With A(BBOX current ) represent at least one candidate frame (e.g., frame BB) obtained by the current action detection. 1 BB 2 and BB 3 ) and the area of ​​this area (for example, the border BB 1 BB 2 and BB 3 The area / size of the area enclosed), BBOX previous With A(BBOX previous ) respectively represent the area enclosed by the previous candidate borders such as the specific border (e.g., border BB0) and the area of ​​this area (e.g., border BB 0 The area / size of the area enclosed by BBOX current ∩BBOX previous With A(BBOX current ∩BBOX previous ) represent at least one candidate border BBOX obtained by the current action detection. current The enclosed area and the previous candidate bounding box BBOX previous The enclosed area (for example, the specific frame such as frame BB 0 The intersection area of ​​the area surrounded by the intersection area) and the area of ​​the intersection area, but the present invention is not limited to this.

[0023] In step 234, the image processing circuit 160 may avoid updating the information of the specific border. For example, when the image processing circuit 160 determines in step 232 that the border BB 1 BB 2 and BB 3 When the object in the frame BB is not the target object, or when the image processing circuit 160 determines in step 233 that the frame BB 1 BB 2 and BB 3 The size is smaller than the frame BB 0 The size and border BB 1 BB 2 and BB 3 Relative to border BB 0 When the overlap ratio is greater than the overlap ratio critical value, the image processing circuit 160 can maintain the frame BB 0 to obtain the complete information of the target object, such as Figure 5 shown.

[0024] In step 235 , the image processing circuit 160 may update the information of the specific frame according to the at least one candidate frame.

[0025] In step 241, the image processing circuit 160 may use the specific frame (eg, frame BB 0 ) to identify the target object.

[0026] In step 242, the image processing circuit 160 may adjust the specific frame (eg, frame BB 0 ) to generate at least one candidate frame, and use the at least one candidate frame to identify the target object to generate the recognition result. For example, the image processing circuit 160 may be based on the frame BB 0 The size and position of the border BB 0 Enlarging by two, four, and eight times to generate multiple candidate bounding boxes having multiple sizes (such as Figure 6 The border BB in the image 600 is shown x2 BB x4 and BB x8 ), and respectively calculate multiple confidence values ​​of the multiple candidate bounding boxes, wherein any confidence value among the multiple confidence values ​​is positively correlated with the probability that the object in the candidate bounding box corresponding to the any confidence value is the target object. For example, the larger the confidence value, the higher the probability that the object in the candidate bounding box corresponding to the confidence value is the target object. It should be noted that the number and value of the above multiple different magnifications do not constitute a limitation to the present invention, and the number and value of the multiple different magnifications can be designed according to the limitation of hardware resources (such as computing power).

[0027] In step 243, the image processing circuit 160 may update the information of the specific frame according to the candidate frame with the largest confidence value among the multiple candidate frames. In actual operation, the closer the size of the candidate frame is to the size of the target object, the higher the confidence value of the candidate frame. Figure 6 In the embodiment shown, the image processing circuit 160 may x8 to update the information for that specific border.

[0028] In step 250, the image processing circuit 160 may output and store the result obtained from the above process (e.g., whether the target object and / or the latest specific border information is detected) to the storage device 140, for example, by transmitting the result back to the relevant program module running on the image processing circuit 160, but the present invention is not limited to this.

[0029] In addition, in step 242 and step 243, the image processing circuit 160 will generate multiple candidate frames with the multiple sizes and then find the candidate frame with the maximum confidence value, but the present invention is not limited to this. In some embodiments, the image processing circuit 160 may sequentially generate the frame BB 0 Magnify twice, four times and eight times, and when the border BB 0 When the image is enlarged to a size of a certain magnification, that is, when the local maximum confidence value is found (for example, when the confidence value obtained by enlarging the image by two times is greater than the confidence value obtained before the enlargement and the confidence value obtained by enlarging the image by four times is less than the confidence value obtained by enlarging the image by two times, the confidence value obtained by enlarging the image by two times is the local maximum confidence value), the image processing circuit 160 can directly use the local maximum confidence value as the global maximum confidence value in order to save computational complexity.

[0030] Although motion detection greatly reduces the amount of computation required by traditional object detection methods, when the object being detected gradually stops moving or becomes stationary, the motion detection methods used in related technologies may mistake the object for the background and cause misjudgment. The object detection method and electronic device 10 provided by the present invention can trigger corresponding mechanisms to find the candidate frame with the highest confidence value when motion is detected and when no motion is detected.

[0031] For example, when the detected target object stops moving, the target object may only move in a part of the area. At this time, the image processing circuit 160 may obtain one or more candidate frames of smaller size, such as Figure 4 However, since the size of the one or more candidate frames is smaller than the size of the candidate frames (e.g., the specific frame) retained when processing the previous image, and the overlap rate between the two exceeds the overlap rate threshold, the image processing circuit 160 may directly retain the specific frame to ensure that the most information can be obtained, such as Figure 5 shown.

[0032] For another example, when the detected target object gradually approaches the electronic device 10, the specific frame can only capture a partial area of ​​the target object, such as Figure 6 The border BB shown 0 The image processing circuit 160 can obtain the area of ​​the frame BB 0 Resize to find the candidate bounding box with the highest confidence value.

[0033] In summary, the object detection method and electronic device provided by the present invention provide corresponding frame adjustment mechanisms for situations with and without motion, respectively, to ensure that in various situations (such as when the object is moving, the object is gradually still, the object is completely still, etc.), it is possible to accurately determine whether the target object appears in the image, and store at least one reliable candidate frame for use in object detection of subsequent input images. Compared with the related art, the embodiments of the present invention do not significantly increase the additional cost, so the present invention can improve the performance of object detection without side effects or with less side effects.

[0034] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

[0035] Description of reference numerals:

[0036] 10: Electronic devices

[0037] 120: Camera module

[0038] 140: Storage device

[0039] 160: Image processing circuit

[0040] 160C: Program code

[0041] IMG, 300, 400, 600: Image

[0042] 210, 220, 231, 232, 233, 234, 235, 241, 242, 243, 250: Steps

[0043] BB 0 BB 1 BB 2 BB 3 BB x2 BB x4 BB x8 :frame

Claims

1. An object detection method, comprising: Using the image processing circuit to determine whether an action occurs in the image to generate a determination result; According to the judgment result, selectively using a specific frame to identify the target object to generate a recognition result, wherein the specific frame represents the position of the target object in the previous image; wherein when the judgment result indicates that no action occurs in the image, using the specific frame to identify the target object; and Selectively updating the information of the specific frame according to the recognition result; The step of using the specific frame to identify the target object includes: adjusting the size of the specific frame to generate at least one candidate frame; and using the at least one candidate frame to identify the target object to generate the identification result; Among them, the step of adjusting the size of the specific border to generate the at least one candidate border includes generating multiple candidate borders having multiple sizes respectively; and using the at least one candidate border to identify the target object to generate the recognition result includes using the image processing circuit to respectively calculate multiple confidence values ​​of the multiple candidate borders, wherein any confidence value among the multiple confidence values ​​is positively correlated with the probability that the object within the candidate border corresponding to the any confidence value is the target object.

2. The object detection method according to claim 1, wherein the step of selectively updating the information of the specific frame according to the recognition result comprises: The information of the specific frame is updated according to the candidate frame having the largest confidence value among the multiple candidate frames.

3. An object detection method, comprising: Using the image processing circuit to determine whether an action occurs in the image to generate a determination result; According to the judgment result, a specific frame is selectively used to identify the target object to generate a recognition result, wherein the specific frame represents the position of the target object in the previous image; wherein, when the judgment result indicates that an action occurs in the image, at least one candidate frame is obtained according to the position of the action in the image, and the target object is identified using the at least one candidate frame to generate the recognition result; and Selectively updating the information of the specific frame according to the recognition result; Among them, selectively updating the information of the specific border based on the recognition result includes: when the recognition result indicates that the object within the at least one candidate border is the target object, selectively updating the information of the specific border based on the size of the at least one candidate border or the overlap rate of the at least one candidate border relative to the specific border.

4. The object detection method according to claim 3, wherein selectively updating the information of the specific frame according to the recognition result comprises: When the recognition result indicates that the object within the at least one candidate bounding box is not the target object, avoiding updating the information of the specific bounding box.

5. The object detection method according to claim 3, wherein selectively updating the information of the specific frame according to the recognition result comprises: When the size of the at least one candidate frame is larger than the size of the specific frame or the overlap rate of the at least one candidate frame relative to the specific frame is smaller than a critical value, the information of the specific frame is updated according to the at least one candidate frame.

6. An electronic device comprising: A camera module, used to acquire images; A storage device, used to store information of a specific frame, wherein the specific frame represents a position of a target object in a previous image; as well as An image processing circuit is coupled to the camera module and the storage device, and is used to execute program code to perform object detection on the image, wherein: The image processing circuit determines whether an action occurs in the image to generate a determination result; According to the judgment result, the image processing circuit selectively uses the specific frame to identify the target object to generate a recognition result; wherein, when the judgment result indicates that no action occurs in the image, the specific frame is used to identify the target object; when the judgment result indicates that an action occurs in the image, at least one candidate frame is obtained according to the position of the action in the image, and the target object is identified using the at least one candidate frame to generate the recognition result; and According to the recognition result, the image processing circuit selectively updates the information of the specific frame. Wherein, when the judgment result indicates that no action occurs in the image, the step of using the specific border to identify the target object includes: adjusting the size of the specific border to generate at least one candidate border; and using the at least one candidate border to identify the target object to generate the recognition result; wherein the step of adjusting the size of the specific border to generate the at least one candidate border includes generating a plurality of candidate borders having a plurality of sizes respectively; and using the at least one candidate border to identify the target object to generate the recognition result includes using the image processing circuit to respectively calculate a plurality of confidence values ​​of the plurality of candidate borders, wherein any confidence value among the plurality of confidence values ​​is positively correlated with the probability that the object within the candidate border corresponding to the any confidence value is the target object; Among them, when the judgment result indicates that an action has occurred in the image, selectively updating the information of the specific border based on the recognition result includes: when the recognition result indicates that the object within the at least one candidate border is the target object, selectively updating the information of the specific border based on the size of the at least one candidate border or the overlap rate of the at least one candidate border relative to the specific border.

Images