Method and electronic device for object detection, computer readable storage medium - Patents.com

Abstract

To provide a method for object detection, an electronic apparatus, and a computer readable storage medium.SOLUTION: The method includes: extracting a plurality of identification features from a plurality of reference images related to a target object; selecting a plurality of selected identification features whose quantity is less than the quantity of the identification features from the plurality of identification features to obtain and store a first feature data set; executing feature extraction processing on a detection target image to obtain a second feature data set in response to receiving the detection target image; and executing similarity determination processing on the detection target image based on the first feature data set and the second feature data set to calculate the quantity of instances of the target object in the detection target image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a method and electronic device for object detection, and to a computer-readable storage medium. [Background technology]

[0002] FIG. 1 shows a conventional process used to detect the number of instances of a target object in an image. In use, when it is required to process a target image (P) to detect the number of instances of a target object contained therein, a first feature extraction module 21 (which may be included in a computer device) performs a first extraction action on each reference image (R) (which may be pre-stored in the computer device and both relate to a target object) to obtain a first feature data set associated with the instances of the target object in the reference image (R). Then, a second feature extraction module 22 (which may be included in the computer device) performs a second extraction action on the target image (P) (which may be received by the computer device) to obtain a second feature data set associated with one or more instances of the target object in the target image (P). Then, a similarity detection module 23 (which may be included in the computer device) performs a detection action based on the first feature data set and the second feature data set, and generates a detection image (P') from the target image (P) in which all detected instances of the target object are marked. Then, a calculation module 24 (which may be included in the computing device) performs calculation actions to determine the quantity of instances of the target object in the sensed image (P') and output the result, thus completing the detection of the quantity of instances of the target object in the image.

[0003] However, each time the conventional process is performed, the first extraction action needs to be performed for each reference image (R), which takes a lot of time, thereby reducing the overall efficiency of the conventional process. Furthermore, in some cases, the instances of the target object in some reference images (R) have almost the same or similar appearance. Therefore, the first extraction action performed for each reference image (R) may not provide additional information about the object, which may lead to additional work involved. Summary of the Invention [Problem to be solved by the invention]

[0004] SUMMARY OF THE PRESENT EMBODIMENT The present invention therefore aims to provide a method that can overcome at least one of the disadvantages of the prior art. [Means for solving the problem]

[0005] According to one embodiment of the present disclosure, the method is performed by a processor included in an electronic device, and includes: a step a) of performing a feature extraction process on a plurality of reference images, each of which is associated with a same target object and contains an instance of the target object, to extract a plurality of identifying features from the plurality of reference images; b) performing a classification process to select a number of selected discrimination features from the discrimination features, the number of selected discrimination features being less than the number of the discrimination features, to obtain a first feature data set associated with the instance of the target object represented by the selected discrimination features, and storing the first feature data set in a storage unit; c) in response to receiving a target image, performing a feature extraction process on the target image to obtain a second feature data set; d) accessing the storage unit to obtain the first feature data set and determining whether the target image includes at least one instance of the target object by performing a similarity determination process on the target image based on the first feature data set and the second feature data set; and e) calculating a quantity of instances of the target object in the target image if it is determined that the target image includes at least one instance of the target object.

[0006] Another object of the present disclosure is to provide an electronic device configured to perform the above method.

[0007] According to one embodiment of the present disclosure, the electronic device for detecting an object includes: a storage unit for storing a plurality of reference images, each of which relates to a same target object and contains an instance of said target object; a first feature extraction module that performs a feature extraction process on the reference image to obtain a plurality of identifying features associated with the target object; a classification module for selecting a plurality of selected discrimination features from the plurality of discrimination features, the selected discrimination features having a quantity less than a quantity of the plurality of discrimination features, to obtain a first feature data set associated with the target object and stored in the storage unit; a second feature extraction module configured to, in response to receiving a target image, perform a feature extraction process on the target image to obtain a second feature data set; a similarity determination module that accesses the storage unit to obtain the first feature data set and receives the second feature data set from the second feature extraction module, and performs a similarity determination process on the target image based on the first feature data set and the second feature data set to determine whether the target image includes at least one instance of the target object; and a calculation module for calculating a quantity of instances of the target object in the target image if the target image includes at least one instance of the target object.

[0008] Another object of the present disclosure is to provide a computer readable non-transitory storage medium containing instructions for causing a processor to perform the above method. [Brief description of the drawings]

[0009] Other features and advantages of the present disclosure will become apparent from the following detailed description of the embodiments, which proceeds with reference to the accompanying drawings, in which it is noted that various features may not be drawn to scale. [Figure 1] FIG. 1 is a schematic diagram illustrating a conventional process for detecting the number of instances of a target object in an image. [Diagram 2] FIG. 2 is a flow chart illustrating steps of a method for object detection according to one embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram illustrating the configuration of an exemplary electronic device that may be used for object detection according to one embodiment of the present disclosure. [Figure 4] FIG. 4 shows a source image used to acquire multiple reference images. [Diagram 5] FIG. 5 illustrates an exemplary process flow involved in obtaining a first feature data set associated with a target object. [Figure 6]FIG. 6 illustrates an exemplary process flow involved in a detection process according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Before describing the present disclosure in more detail, it should be noted that, where considered appropriate, reference numerals or trailing portions of reference numerals may be repeated among the figures to indicate corresponding or similar elements, which may optionally have similar features.

[0011] Throughout this disclosure, the terms "coupled to" or "connected to" may refer to a direct connection between two or more electrical apparatuses / devices / instruments via a conductive material (e.g., electrical wires) or an indirect connection between two electrical apparatuses / devices / instruments via one or more other apparatuses / devices / instruments or wireless communication.

[0012] 2 is a flow chart illustrating steps of a method for object detection according to one embodiment of the present disclosure, in this embodiment, the method is performed using an electronic device.

[0013] 3 is a block diagram showing an example of the configuration of an electronic device 1 used for object detection in one embodiment of the present disclosure. In this embodiment, the electronic device 1 may be, for example, a server, a computer, a laptop, a tablet, or a mobile device, and includes a storage unit 11, a processor 12, and a communication unit 13.

[0014] The storage unit 11 may be implemented using, for example, a random access memory (RAM), a read only memory (ROM), a programmable ROM (PROM), firmware, or flash memory, etc. The storage unit 11 stores a software application and a number of reference images P1.

[0015] The processor 12 is connected to the storage unit 11 and may include, but is not limited to, a single-core processor, a multi-core processor, a dual-core mobile processor, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), etc. The software application includes instructions that, when executed by the processor 12, cause the processor 12 to perform the following operations: In the embodiment shown in Fig. 3, the processor 12 executing different parts of the software application may operate as various modules including a classification module 121, a first feature extraction module 122, a second feature extraction module 123, a similarity determination module 124, and a calculation module 125.

[0016] It is noted that in other embodiments, the modules included in the processor 12 can be implemented using one or more Application Specific Integrated Circuit (ASIC) chips, one or more Programmable Logic Devices (PLDS), or one or more microcontrollers each having firmware embedded therein. Each module included in the processor 12 can be configured with a neural network model to perform the associated processing. For example, in some embodiments, the first feature extraction module 122 and the second feature extraction module 123 can both be implemented using a You Only Look Once (YOLO) neural network model, using a Convolutional Neural Network (CNN) structure as a backbone. The structure (layers, neurons, and associated parameters) of the YOLO neural network model that constitutes the first feature extraction module 122 can be similar or identical to that of the second feature extraction module 123, and thus in some embodiments, the first feature extraction module 122 and the second feature extraction module 123 can be implemented using two identical modules. It is noted that in different applications, the first feature extraction module 122 and the second feature extraction module 123 can include different neural networks with different structures.

[0017] The communications unit 13 is connected to the processor 12 and may include one or more of a radio frequency integrated circuit (RFIC), a short-range wireless communications module supporting a short-range wireless communications network using wireless technologies such as Bluetooth or Wi-Fi, a mobile communications module supporting long-range telecommunications using wireless mobile long-range telecommunications technologies such as Long Term Evolution (LTE), third generation (3G), fourth generation (4G), fifth generation (G), or the like.

[0018] The multiple reference images P1 are all related to the same target object. In the example shown in FIG. 4 where the source image (A) only contains strawberries, the target object is strawberry. However, the present disclosure is not limited to such an example. In this embodiment, the multiple reference images P1 can be pre-acquired from the source image (A) (pre-stored in the storage means 11 or acquired by the communication unit 13) by the processor 12. Specifically, multiple different areas in the source image (A) each contain a recognizable instance of the target object, and can be identified and cropped out to become the reference image P1. In some embodiments, the target object is identified manually, while in other embodiments, the target object is automatically identified by an algorithm that identifies the object, and may differ from the result of the manual identification.

[0019] In some embodiments, before the method is performed, the processor 12 performs a data enlargement process on one or more reference images P1 to generate at least one enlarged image as an additional reference image P1. The data enlargement process may include one or more of the following operations: flipping (vertical or horizontal), rotating, cropping, and resizing (enlarging or reducing). Note that the data enlargement process is well known in the relevant technical field, and therefore will not be described in detail for the sake of brevity.

[0020] In use, when it is required to perform an object detection process on a detected object image P3 to determine the number of target objects contained in the detected object image P3 which are recognised as instances of target objects, it is conceivable that the method shown in Figure 2 would be performed using the electronic device 1 shown in Figure 3.

[0021] In step S1, the processor 12 extracts a number of identifying features from a number of reference images P1. In some embodiments, each identifying feature is associated with a unique appearance-related feature (e.g., shape, material, etc.) of the target object, but is not limited thereto.

[0022] Specifically, these reference images P1 are input to a first feature extraction module 122 of the processor 12. In response, the first feature extraction module 122 performs a feature extraction process on the reference images P1 to output a number of discriminative features, which are then transferred to the classification module 121 for further processing.

[0023] In step S2, the classification module 121 is configured to execute a classification process in response to receiving these discrimination features, to select a plurality of selected discrimination features P2 from the discrimination features received from the first feature extraction module 122, and to generate a first feature data set using these selected discrimination features. Note that the number of selected discrimination features P2 is less than the number of discrimination features.

[0024] Specifically, in some embodiments, the first feature extraction module 122 is configured to assign, for each identifying feature, one of a number of groups based on the appearance-related features of the target object instances to which the identifying feature is associated. Each group can be, but is not limited to, associated with one appearance-related feature possessed by the target object. The classification module 121 is configured to determine, for each identifying feature, the appearance-related features of the target object instances based on the identifying feature, and assign the identifying feature to one group based on the appearance-related features thus determined. Thus, the target object instances represented by identifying features that are assigned to the same group all have similar appearances. Note that this classification process is well known in the relevant technical field, and therefore will not be described in detail for the sake of brevity.

[0025] The classification module 121 is then configured to select one discrimination feature from each group as the selected discrimination feature P2. Thus, a number of selected discrimination features P2 are selected from the multiple groups, respectively. Thus, subsequent processing can be performed on the selected discrimination features P2 instead of all discrimination features extracted from the reference image P1, and thus in this manner the number of required calculations can be reduced without sacrificing the diversity of information contained in the reference image P1. As a result, each selected discrimination feature P2 is associated with a different appearance-related feature of the target object. In other words, discrimination features in each reference image P1 that show the same appearance-related feature of the target object as that shown by one of each selected discrimination feature P2 are discarded, so that the number of selected discrimination features P2 is less than the number of discrimination features extracted by the first feature extraction module 122. Note that in other embodiments, the selected discrimination features P2 can be selected in different ways, and are not limited to the above. The selected discrimination features P2 obtained by the classification module 121 constitute a first feature data set stored in the storage unit 11.

[0026] An exemplary flow chart of the processes involved in steps S1 and S2 is shown in Fig. 5. In some embodiments, the content of the first feature data set may include, for example, appearance-related features (e.g., shape, material, etc.) of many instances of the target object depicted in the selected discriminatory features P2. Thus, the size of the first feature data set may be reduced without compromising the information diversity of the data from the reference image P1.

[0027] 6, after the first feature data set is generated and stored, a detection target image P3 obtained from the storage means 11 or from a remote device via a network (e.g., the Internet) is input to the second feature extraction module 123. The processor 12 is configured to perform a detection process including steps S3 to S5 in response to receiving the detection target image P3. Specifically, in step S3, for each object detected in the detection target image P3 (hereinafter referred to as the “detected object”), the second feature extraction module 123 performs a feature extraction process on the object to obtain a second feature data set related to the detected object, and then transfers the obtained second feature data set to the similarity determination module 124.

[0028] In step S4, the similarity determination module 124 accesses the storage unit 11 to obtain the first feature data set, and performs a similarity determination process on the detection target image P3 based on the first feature data set and the second feature data set to determine whether the detection target image P3 contains at least one instance of the target object. Specifically, the similarity determination process calculates a similarity score between an object represented by the second feature data set (e.g., one detected object included in the detection target image P3) and an object represented by the first feature data set for each selected identification feature included in the first feature data set (e.g., each representing a unique appearance-related feature of the target object). Note that the process of calculating the similarity score is well known in the relevant technical field, and therefore will not be described in detail for the sake of brevity.

[0029] If a similarity score exceeding a predetermined threshold is obtained, the similarity determination module 124 determines that the object included in the detection target image P3 is an instance of the target object.

[0030] In practice, the similarity determination module 124 may determine one or more detected objects as instances of the target object (hereinafter, these detected objects are referred to as identified objects).

[0031] In some embodiments, before the similarity determination process, the similarity determination module 124 first performs a normalization process on the second feature data set, because different detected object images may have different sizes, and therefore the second feature data set compared with the first feature data set is normalized.

[0032] In some embodiments, the similarity determination module 124 is further configured to mark each identified object in the target image P3 to generate a marked image P4. In some embodiments, the marks can be dots having a relatively light color.

[0033] On the other hand, if no similarity score is determined that exceeds the predetermined threshold, it can be assumed that the detection target image P3 does not contain an instance of the object (i.e., none of the detected objects in the detection target image P3 are the target object), and the similarity determination module 124 can output a signal indicating that no target object has been detected, in which case the method is immediately terminated.

[0034] If not, and at least one detected object in the detection target image P3 is determined to be an instance of the target object, a marked image P4 is generated based on the detection target image P3 and transferred to the calculation module 125.

[0035] In step S5, the calculation module 125 calculates the number of identified objects in the detection target image P3 based on the marked image P4 (i.e., calculates the number of instances of the target object in the detection target image P3), and outputs the result. Specifically, the calculation module 125 can include a peak detection algorithm (e.g., a "peak_local_max" function) that is applied to the marked image P4 to identify the marks made by the similarity determination module 124 in the marked image P4. In this way, the calculation module 125 is configured to record the number of marks found in the marked image P4, which is the same as the number of identified objects in the detection target image P3. Then, the number of identified objects in the detection target image P3 can be recorded or output as a result of the number of instances of the target object in the detection target image P3, and the above method is completed.

[0036] In this embodiment, the calculation of the number of identified objects in the detection target image P3 is performed by first generating a marked image P4 and then processing the marked image P4 to count the number of identified objects, but in other embodiments, the calculation of the number of identified objects may be performed in other ways (e.g., using a heat map) and is not limited to the above embodiment.

[0037] It should be noted that the above method may have the following advantages over the conventional process. First, since the processor 12 is configured to select a plurality of selected identification features P2 from a plurality of identification features extracted by the processor 12 from the reference image P1, and the selection is performed such that the number of selected identification features P2 is less than the identification features extracted by the processor 12, and the selected identification features P2 still hold various instances of the target object, a relatively small number of images can be processed, which increases efficiency and does not sacrifice the diversity of information contained in the first feature data set for subsequent processing. Furthermore, after obtaining the first feature data set related to the target object, the first feature data set is stored in the data storage 11 and can be accessed when processing is performed to detect the same object for an image. In other words, when the above method is performed later for the same target object, steps S1 and S2 can be omitted, and only steps S3 to S5 in the detection process need to be performed. Therefore, when the above method is performed later for the same target object, the calculations can be greatly reduced.

[0038] In some embodiments, if an additional image of the target object is to be provided as a reference image P1 (to provide additional information to the first feature dataset) after one execution of the method is completed, steps S1 and S2 or similar steps S1 and S2 can be performed after receiving one or more additional images (as one or more additional reference images P1) to update the first feature dataset and store the updated first feature dataset in the storage unit 11. That is, the first feature extraction module 122 can obtain other discriminatory features by performing a feature extraction process on the added reference image, and the classification module 121 can update the first feature dataset based on the other discriminatory features (which may or may not be selected as other discriminatory features) and store the updated first feature dataset in the storage unit 11.

[0039] In some embodiments, this additional image is calculation The additional images may be images cropped from the detection target image P3 by module 125, each of which includes one recognizable instance of the target object. For example, after one execution of the method, the detection target image P3 may be manually inspected to obtain a plurality of cropped images, each of which includes one recognizable instance of the target object. Alternatively, the processor 12 may be configured to crop a portion of the detection target image P3 that includes an object whose similarity score exceeds a predetermined threshold to obtain the cropped images.

[0040] In summary, the embodiments of the present disclosure provide a method and an electronic device for performing object detection. In the method, a plurality of reference images, each of which includes a recognizable instance of a target object, are processed, and each reference image is assigned to a group based on the appearance-related features of the instance of the target object contained in the reference image. Then, a plurality of selected images are selected from the reference images, so as to ensure the diversity of the appearance-related features of the instance of the target object contained in the selected images, and the number of selected images is less than the number of reference images. Then, a first feature data set is obtained by processing the selected images, and the first feature data set is related to the target object and is used when processing a target image to determine the number of instances of the target object in the target image. Thus, the process of obtaining the first feature data set related to the target object can be achieved by processing a small number of images, thereby improving the efficiency of the process without compromising the diversity of the processed data. In some cases, as the number of images processed is reduced, the structure of the neural network model used to extract the features can be modified to allow more features to be extracted from each selected image, thus ultimately improving the accuracy of object detection.

[0041] According to one embodiment of the disclosure, there is provided a computer readable non-transitory storage medium for implementing the method of detecting an object, the computer readable non-transitory storage medium storing a software application including instructions that, when executed by a processor of an electronic device, cause the processor to perform the operations described in the method steps of Figure 2. In some embodiments, the computer readable non-transitory storage medium can be in a form similar to the data storage 11 in the electronic device 1.

[0042] In the above description, for the purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that one or more other embodiments may be practiced without some of these specific details. It should also be understood that throughout this specification, references to "one embodiment," "one embodiment," an embodiment with an ordinal number, and the like, mean that certain features, structures, or characteristics may be included in the implementation of the present disclosure. It should further be understood that various features may be grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of various inventive aspects, and that one or more features or specific details from one embodiment may be implemented with one or more features or specific details from another embodiment, as necessary, in the implementation of the present disclosure.

[0043] While the present disclosure has been described in connection with what are considered to be the exemplary embodiments, it is understood that the disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements within the broadest spirit and scope so as to encompass all such modifications and equivalent arrangements.

Claims

1. 1. A method for object detection executed by a processor of an electronic device, comprising: a) performing a feature extraction process on a plurality of reference images, each of which is associated with a same target object and contains an instance of the target object, to extract a plurality of identifying features from the plurality of reference images; b) performing a classification process to select from the discriminatory features a number of selected discriminatory features, the number of selected discriminatory features being less than the number of the discriminatory features, to obtain a first feature data set associated with the instance of the target object represented by the selected discriminatory features, and storing the first feature data set in a storage unit; In response to receiving a target image, performing a feature extraction process on the target image to obtain a second feature data set; and a step d of accessing the storage unit to obtain the first feature data set and determining whether the target image includes at least one instance of the target object by performing a similarity determination process on the target image based on the first feature data set and the second feature data set; e., if it is determined that the target image contains at least one instance of the target object, calculating a quantity of instances of the target object in the target image.

2. The method of claim 1 , further comprising performing steps a and b to update the first feature data set when an additional image is received as another reference image.

3. The method of claim 2 , wherein the additional image is cropped from the target image and includes an instance of the target object.

4. Before step a, performing a data enlargement process on one or more of the reference images to generate at least one enlarged image as another reference image; The method of claim 1 , wherein the data upscaling operation includes one or more of a vertical flip operation, a horizontal flip operation, a rotation operation, a crop operation, and a resize operation.

5. In step a, each of the discrimination features is related to an appearance-related feature of the target object; The method of claim 1 , wherein in step b, each of the selected discriminatory features is associated with one unique appearance-related feature of the target object.

6. In the step b, for each said identifying feature, assigning said identifying feature to one of a plurality of groups based on appearance-related features of the instance of the target object represented by said identifying feature; and selecting one of the distinguishing features from each of the groups to be one of a plurality of the selected distinguishing features.

7. 2. The method of claim 1, wherein step e includes: generating a marked image based on the detection target image, the marked image having each of the at least one instance of the target object marked; and calculating a quantity of instances of the target object in the detection target image based on the marked image.

8. a storage unit for storing a plurality of reference images, each of which relates to a same target object and contains an instance of said target object; a first feature extraction module for performing a feature extraction process on the reference image to obtain a plurality of identifying features associated with the target object; a classification module for selecting a plurality of selected discriminatory features from the plurality of discriminatory features, the selected discriminatory features having a quantity less than a quantity of the plurality of discriminatory features, to obtain a first feature data set associated with the target object and stored in the storage unit; a second feature extraction module configured to receive a target image and perform a feature extraction process on the target image to obtain a second feature data set; a similarity determination module that accesses the storage unit to obtain the first feature data set and receives the second feature data set from the second feature extraction module, and determines whether the detection target image includes at least one instance of the target object by performing a similarity determination process on the detection target image based on the first feature data set and the second feature data set; and a calculation module for calculating a quantity of instances of the target object in the target image if the target image includes at least one instance of the target object.

9. 9. The electronic device of claim 8, wherein when an additional image is received as another reference image, the first feature extraction module performs a feature extraction process on the other reference image to obtain other identifying features, and the classification module updates the first feature dataset based on the other identifying features and then stores the updated first feature dataset in the storage unit.

10. The electronic device of claim 9 , wherein the additional image is cropped from the sensed target image by the computing module and includes an instance of the target object.

11. a processor for performing a data enlargement process on one or more of the reference images to generate at least one enlarged image as another reference image; The electronic device of claim 8 , wherein the data enlargement operation includes one or more of a vertical flip operation, a horizontal flip operation, a rotation operation, a crop operation, and a resize operation.

12. 9. The electronic device of claim 8, wherein the first feature extraction module extracts each of the discriminatory features as being associated with an appearance-related feature of the target object, and the classification module selects each of the selected discriminatory features as being associated with one unique appearance-related feature of the target object.

13. The classification module includes: for each said distinguishing feature, assigning said distinguishing feature to one of a plurality of groups based on appearance-related features of the instance of the target object represented by said distinguishing feature; 13. The electronic device of claim 12, further comprising: selecting one said distinguishing feature from each said group to be one of a plurality of said selected distinguishing features.

14. If the detection target image includes at least one instance of the target object, the similarity determination module generates a marked image based on the detection target image, the marked image having each of the at least one instance of the target object marked; The electronic device of claim 13 , wherein the calculation module calculates a quantity of instances of the target object in the detection target image based on the marked image.

15. A computer-readable non-transitory storage medium comprising instructions that, when executed by a processor of an electronic device, cause the processor to perform each step of the method recited in claim 1.

Images