Object recognition device and object recognition method

Abstract

The invention provides an object recognition apparatus and an object recognition method. The object recognition method comprises the following steps: receiving a first image and receiving user input; detecting an object of the first image to obtain a detection area and an object category; in response to matching of the user input and the detection area, generating an identification result of the object through a first machine learning model corresponding to the object category; and outputting information corresponding to the recognition result.

Description

Technical Field

[0001] The present disclosure relates to an object recognition device and an object recognition method. Background Art

[0002] Object recognition is the technique of locating and identifying objects present in an image. Since images may contain random objects, object recognition models for images should be able to recognize different kinds of objects. However, the recognition results produced by such object recognition models may lack a lot of detailed information. For example, an object recognition model trained for various object categories may only be able to locate people in an image, while an object recognition model dedicated to facial recognition can identify the identity of people in an image. Therefore, how to accurately provide a method that can recognize various object categories is one of the important problems in the field. Summary of the invention

[0003] The present disclosure relates to an object recognition device and an object recognition method, by which detailed information of an object in an image is accurately recognized.

[0004] The object recognition device of the present invention comprises a first transceiver, an input device, an output device and a processor. The first transceiver receives a first image. The input device receives a user input. The processor is coupled to the input device, the output device and the first transceiver and is configured to: detect an object of the first image to obtain a detection area and an object category; in response to matching the user input with the detection area, generate a recognition result of the object through a first machine learning model corresponding to the object category; and output information corresponding to the recognition result through the output device.

[0005] In one embodiment of the present invention, the input device includes an image capture device, and the processor is further configured to: obtain a second image through the image capture device; and perform eye tracking on the second image to obtain user input.

[0006] In one embodiment of the present invention, the object recognition device further includes a sound capture device coupled to the processor and obtaining an audio signal, wherein the processor generates a recognition result through a first machine learning model in response to the audio signal.

[0007] In one embodiment of the present invention, the output device includes a display device, and the processor is further configured to: display a graphical user interface through the output device, wherein the graphical user interface includes information, wherein the information includes at least one of descriptive information, personal profile, summary information or representative diagram of the object.

[0008] In one embodiment of the present invention, the output device comprises a second transceiver, and the processor is further configured to: transmit the access command corresponding to the information through the output device.

[0009] In one embodiment of the present invention, the output device comprises a display device for displaying the first image, wherein the processor is further configured to: in response to the user input matching the detection area, highlight the object of the first image through the display device.

[0010] In one embodiment of the present invention, the processor is further configured to: access an external database through the first transceiver; and obtain information from the external database according to the recognition result.

[0011] In one embodiment of the present invention, the object recognition device further includes a storage medium coupled to the processor and storing a database, wherein the processor is further configured to: obtain information from the database according to the recognition result.

[0012] In one embodiment of the present invention, the object recognition device also includes a storage medium coupled to a processor and storing multiple machine learning models, wherein the multiple machine learning models include a first machine learning model, wherein the processor selects the first machine learning model from the multiple machine learning models based on the object category and generates a recognition result based on the selected first machine learning model.

[0013] In one embodiment of the present invention, the first machine learning model includes a You Only Look Once (YOLO) model.

[0014] In one embodiment of the present invention, the processor is further configured to: detect the object of the first image by using an object detection model to obtain a detection area and an object category.

[0015] In one embodiment of the present invention, the object category includes one of the following: identification of the object, text, two-dimensional barcode, or product.

[0016] The object recognition method of the present invention includes: receiving a first image and receiving user input; detecting the object of the first image to obtain a detection area and an object category; in response to matching the user input with the detection area, generating a recognition result of the object through a first machine learning model corresponding to the object category; and outputting information corresponding to the recognition result.

[0017] Based on the above, the object recognition device of the present invention can generate detailed information of only the object of the image that the user is interested in. Therefore, a large amount of computing resources used for object recognition can be saved.

[0018] In order to make the foregoing content more understandable, several embodiments are described in detail below with accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are included to provide further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] Figure 1 A schematic diagram showing an object recognition device according to an embodiment of the present invention;

[0021] Figure 2 A flowchart showing an object recognition method according to an embodiment of the present invention;

[0022] Figure 3 A schematic diagram showing an image according to an embodiment of the present invention;

[0023] Figure 4 A flow chart of an object recognition method according to an embodiment of the present invention is shown.

[0024] Description of Figure Numbers

[0025] 10: Object recognition device;

[0026] 20: Image;

[0027] 30, 60: objects;

[0028] 40: bounding box;

[0029] 50: Graphical user interface;

[0030] 100: processor;

[0031] 200: storage medium;

[0032] 210: Object detection model;

[0033] 220: Machine learning models;

[0034] 230: Database;

[0035] 300: transceiver;

[0036] 400: input device;

[0037] 500: output device;

[0038] 600: sound capturing device;

[0039] S201, S202, S203, S204, S205, S206, S207, S208, S209, S401, S402, S403, S404: steps. DETAILED DESCRIPTION

[0040] Figure 1A schematic diagram of an object recognition device 10 according to an embodiment of the present invention is shown. The object recognition device 10 may include a processor 100, a storage medium 200, a transceiver 300, an input device 400, an output device 500, and a sound capture device 600. The processor 100 may be connected to the storage medium 200, the transceiver 300, the input device 400, the output device 500, and the sound capture device 600.

[0041] The processor 100 is, for example, a central processing unit (CPU), or another programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or other similar components, or a combination thereof. The processor 100 is capable of accessing and executing various modules or applications stored in the storage medium 200 to perform the functions of the object recognition device 10.

[0042] The storage medium 200 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD) or similar element, or a combination thereof, configured to record a plurality of modules or applications executable by the processor 100. In the present embodiment, the storage medium 200 may store an object detection model 210, one or more machine learning models 220, and a database 230.

[0043] The transceiver 300 transmits or receives signals wirelessly or by wire. The transceiver 300 may also perform operations such as low noise amplification (LNA), impedance matching, mixing, up / down conversion, filtering, amplification, and the like.

[0044] The input device 400 can be manipulated by a user to receive user input. In one embodiment, the input device 400 can include an image capture device, wherein the image capture device can include an image sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD).

[0045] The output device 500 may output information such as images, signals or sounds of a user. In one embodiment, the output device 500 may include a display device or a transceiver. The display device includes, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, a vacuum fluorescent display (VFD), a plasma display panel (PDP), an organic light-emitting diode (OLED) or a field-emission display (FED).

[0046] The sound capturing device 600 is, for example, a condenser microphone, an electret condenser microphone (ECM), or a micro-electro-mechanical system (MEMS) microphone.

[0047] Figure 2 A flow chart of an object recognition method according to an embodiment of the present invention is shown, wherein the object recognition method may be performed as follows: Figure 1 The object recognition device 10 shown in FIG.

[0048] In step S201, the processor 100 may receive an image to be recognized through the transceiver 300, wherein the image may be an image displayed by the output device 500. For example, the image may be a video of a basketball game currently being watched by a user.

[0049] In step S202 , the processor 100 may perform pre-processing on the image, wherein the pre-processing may include but is not limited to noise reduction, contrast enhancement, image resizing, color correction, segmentation, or feature extraction.

[0050] In step S203, the processor 100 may detect the object of the image to obtain a detection area and an object category. The processor 100 may detect the object of the image to obtain a detection area and an object category through the object detection model 210, wherein the detection area may represent the location of the object of the image. The object category may include but is not limited to the identification of an object (e.g., a living organism, such as a person, an animal, a plant, an insect, or a bacterium), text, a two-dimensional barcode (e.g., a quick response (QR) code), or a product (e.g., a natural product, such as a diamond or a pearl, or an artificial product, such as a vehicle, furniture, or a tool). In one embodiment, the object detection model 210 may include a machine learning model, such as a convolution neural network (CNN) model. Figure 3 A schematic diagram of an image according to one embodiment of the present invention is shown. For example, the processor 100 may detect an object 30 in an image 20 to obtain a detection area and an object category of the object 30. Assuming that the object 30 is a person (i.e., the object category of the object 30 is) "person", the processor 100 may obtain a detection area representing the position of the person in the image 20, wherein the detection area may be an area surrounded by a bounding box 40 generated by the object detection model 210. For another example, the processor 100 may detect an object 60 in the image 20 to obtain a detection area and an object category of the object 60. Assuming that the object 60 is a two-dimensional barcode (i.e., the object category of the object 60 is "two-dimensional barcode"), the processor 100 may obtain a detection area representing the position of the two-dimensional barcode in the image 20.

[0051] Return to reference Figure 2 In step S204, the processor 100 may receive user input through the input device 400. For example, assuming that the input device 400 is an image capture device. The processor 100 may obtain an image of a user who is viewing an image displayed by the output device 500. The processor 100 may perform eye tracking on the image of the user to obtain the user input, wherein the user input may be a gaze position of the user on the image displayed by the output device 500.

[0052] In step S205, the processor 100 may determine whether the user input matches the detection area of ​​the object (e.g., object 30 or object 60). That is, the processor 100 may determine whether the gaze position of the user overlaps with the detection area of ​​the object (i.e., whether the user gazes at the object on the image displayed by the output device 500). If the user input matches the detection area of ​​the object (i.e., the user gazes at the object), the processor 100 may determine that the detection area of ​​the object is a region of interest (ROI), and then proceed to step S206. If the user input does not match the detection area of ​​the object (i.e., the user does not gaze at the object), the processor 100 may perform step S204 again.

[0053] In one embodiment, the processor 100 may highlight an object (e.g., object 30 or object 60) in the image 20 through the output device (i.e., display device) 500 in response to the user input matching the detection area of ​​the object. For example, when the user is not gazing at the object 30, the processor 100 may hide the bounding box 40 of the object 30 on the image 20. However, after the processor 100 determines that the user is gazing at the object 30, the processor 100 may show the bounding box 40 on the image 20 to highlight the object 30.

[0054] In step S206, the processor 100 may determine whether the object recognition device 10 is in real-time mode. If the object recognition device 10 is in real-time mode, the processor 100 may proceed to step S207. If the object recognition device 10 is not in real-time mode, the processor 100 may proceed to step S208. In one embodiment, the object recognition device 10 may enter or leave the real-time mode according to a configuration from the user. For example, the processor 100 may receive a user command through the transceiver 300 and determine whether to enter or leave the real-time mode according to the user command.

[0055] In step S207, the processor 100 may determine whether the user authorization is received by the processor 100. If the user authorization is received by the processor 100, the processor 100 may proceed to step S207. If the user authorization is not received by the processor 100, the processor 100 may execute step S207 again to wait for the user authorization. In one embodiment, the user authorization may include an audio signal, wherein the processor 100 may obtain the audio signal through the sound capture device 600. For example, if the processor 100 captures the sound of a sentence "show details" spoken by the user, the processor 100 may determine that the user authorization is received by the processor 100.

[0056] In step S208, the processor 100 may generate a recognition result of an object (e.g., object 30 or object 60) through a machine learning model 220 corresponding to an object category of the object. Specifically, the storage medium 220 may store a plurality of machine learning models 220 trained for different object categories, respectively. The processor 100 may select a machine learning model 220 from a plurality of machine learning models 220 according to the object category, and the processor 100 may generate a recognition result according to the selected machine learning model 220. For example, in response to determining that the object category of the object 30 is "person", the processor 100 may select a machine learning model 220 trained for the object category "person" from a plurality of machine learning models 220, and then the processor 100 may input the detection area of ​​the object 30 into the selected machine learning model 220. The selected machine learning model 220 may output a recognition result of the object 30, wherein the recognition result of the object 30 may indicate the identity of the person. For another example, in response to determining that the object category of the object 60 is “two-dimensional barcode”, the processor 100 may select a machine learning model 220 trained for the object category “two-dimensional barcode” from a plurality of machine learning models 220, and then the processor 100 may input the detection area of ​​the object 60 into the selected machine learning model 220. The selected machine learning model 220 may output a recognition result of the object 60, wherein the recognition result of the object 60 may include a uniform resource locator (URL).

[0057] In one embodiment, the one or more machine learning models 220 stored in the storage medium 200 may include a You Only Look Once (YOLO) model trained to perform, for example, facial detection of living organisms, optical character recognition or semantic segmentation of text, decoding of two-dimensional barcodes, or object recognition of products.

[0058] In step S209, the processor 100 may output information corresponding to the recognition result through the output device 500. In one embodiment, the information may be pre-stored in the database 230 in the storage medium 200 or in an external database (e.g., a cloud server). The processor 100 may access the database 230 or access the external database through the transceiver 300 to obtain information corresponding to the recognition result. For example, the database 230 may pre-store multiple profiles of people. After the processor 100 obtains the recognition result of the object 30 (i.e., the identity of the person), the processor 100 may query the database 230 to obtain the profile of the person corresponding to the recognition result or the object 30.

[0059] In one embodiment, the output device 500 may include a display device. The processor 100 may display a graphical user interface (GUI) through the output device 500, wherein the graphical user interface may include information of the object, such as description information, personal profile, summary information, or a representative diagram of the object. Figure 3 As an example, processor 100 may display graphical user interface 50 on image 20, where graphical user interface 50 may show descriptive information of object 30. For example, if object 30 is a basketball player, graphical user interface 50 may show statistical information of the player.

[0060] In one embodiment, the output device 500 may include a transceiver. The processor 100 may transmit an access command corresponding to the information of the object through the output device 500. Figure 3 As an example, the processor 100 may transmit an access command corresponding to a URL through the output device 500 so that a website corresponding to the URL may be displayed on the image 20 , wherein the URL may be a decoding result of the object 60 .

[0061] Figure 4 A flow chart of an object recognition method according to an embodiment of the present invention is shown, wherein the object recognition method may be performed as follows: Figure 1 The object recognition device 10 shown in is implemented. In step S401, a first image is received and a user input is received. In step S402, an object of the first image is detected to obtain a detection area and an object category. In step S403, in response to the user input matching the detection area, a recognition result of the object is generated by a first machine learning model corresponding to the object category. In step S404, information corresponding to the recognition result is output.

[0062] Based on the above, the present invention can locate and identify the object of the image through a two-stage mechanism. First, the object recognition device of the present invention can detect the image through the object detection model to obtain the detection area and object category of the object of the image. Then, the object recognition device can determine whether the user is interested in obtaining detailed information of the detected object. If the user is interested in obtaining detailed information, the object recognition device can generate the recognition result of the object through the machine learning model. Since the machine learning model is only used to identify the object of interest to the user, a large amount of computing resources used for object recognition can be saved.

[0063] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In view of the foregoing, it is intended that the present disclosure encompass modifications and variations provided that the modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. An object recognition device, characterized in that: include: A first transceiver receives a first image; An input device for receiving user input; Output device; as well as a processor, coupled to the input device, the output device, and the first transceiver, and configured to: detecting an object in the first image to obtain a detection area and an object category; In response to the user input matching the detection area, generating a recognition result of the object by using a first machine learning model corresponding to the object category; as well as Information corresponding to the recognition result is outputted through the output device.

2. The object recognition device of claim 1, wherein the input device comprises an image capture device and the processor is further configured to: obtaining a second image by the image capture device; and Eye tracking is performed on the second image to obtain the user input.

3. The object recognition device according to claim 1, further comprising: A sound capture device is coupled to the processor and obtains an audio signal, wherein The processor generates the recognition result through the first machine learning model in response to the audio signal.

4. The object recognition device according to claim 1, wherein the output device comprises a display device, and the processor is further configured to: A graphical user interface is displayed through the output device, wherein the graphical user interface includes the information, wherein the information includes at least one of the descriptive information, personal profile, summary information, or representative diagram of the object.

5. The object recognition device according to claim 1, wherein the output device comprises a second transceiver, and the processor is further configured to: An access command corresponding to the information is transmitted via the output device.

6. The object recognition device according to claim 1, wherein the output device comprises a display device for displaying the first image, wherein the processor is further configured to: In response to the user input matching the detection area, the object of the first image is highlighted via the display device.

7. The object recognition device according to claim 1, wherein the processor is further configured to: accessing an external database via the first transceiver; and The information is obtained from the external database according to the recognition result.

8. The object recognition device according to claim 1, further comprising: a storage medium coupled to the processor and storing a database, wherein the processor is further configured to: The information is obtained from the database according to the recognition result.

9. The object recognition device according to claim 1, further comprising: A storage medium coupled to the processor and storing a plurality of machine learning models, wherein the plurality of machine learning models include the first machine learning model, wherein The processor selects the first machine learning model from the plurality of machine learning models according to the object category and generates the recognition result according to the selected first machine learning model.

10. The object recognition apparatus of claim 1, wherein the first machine learning model comprises a you-only-see-once model.

11. The object recognition device according to claim 1, wherein the processor is further configured to: The object of the first image is detected by an object detection model to obtain the detection area and the object category. 12 . The object recognition device according to claim 1 , wherein the object category comprises one of the following: recognition of the object, text, a two-dimensional barcode, or a product.

13. An object recognition method, characterized in that: include: receiving a first image and receiving user input; detecting an object in the first image to obtain a detection area and an object category; In response to the user input matching the detection area, generating a recognition result of the object by using a first machine learning model corresponding to the object category; as well as Information corresponding to the recognition result is output.

Images