Object detection device and method

Abstract

To provide an object detection device.SOLUTION: There is provided an object detection device. The object detection device has an image extraction device and a processor. The image extraction device extracts a plurality of images of a specific space and a plurality of depth images, and each of the images corresponds to one of the depth images. The processor detects an object in each image using an object detection model to obtain object information. The processor calculates a depth value of each object in each depth image. The processor accepts input of the depth value of each object to an object state model to determine an object state of the object. The processor determines whether to enter the specific space based upon the object information and object states of the respective objects. The object detection device can therefore determine proper time to enter the specific space.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an object-detection technology, and more particularly to an object-detection technology in which an object detection device determines whether or not an object is entering a specific space. [Background technology]

[0002] With the development of technology, autonomous mobile robots (AMRs) are being applied more widely.

[0003] When an AMR needs to be applied to a product delivery service (e.g., automated food delivery) in a building, the products need to be transported to each floor by elevator. However, before the AMR enters the elevator, a wheelchair, a stroller, or a shopping cart may already be inside the elevator. Due to the large size of the AMR, if the AMR waits directly in front of the elevator door or enters the elevator directly, large objects (e.g., wheelchairs or strollers) cannot smoothly exit the elevator.

[0004] Therefore, how to help AMRs determine the appropriate time to enter an elevator is a topic for discussion. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide an object detection device and method that solves the above problems. [Means for solving the problem]

[0006] One embodiment of the present invention provides an object detection device. The object detection device includes an image extraction device and a processor. The image extraction device extracts a plurality of images of a specific space and a plurality of depth images, each of which corresponds to one of the depth images. The processor is coupled to the image extraction device and obtains the plurality of images and the depth images from the image extraction device. The processor detects an object in each image using an object detection model to obtain object information. The processor calculates a depth value of each object in each depth image. The processor inputs the depth value of each object into an object state model to determine an object state of each object. The processor determines whether to enter the specific space based on the object information and the object state of each object.

[0007] One embodiment of the present invention provides an object detection method. The object detection method is applied to an object detection device. The object detection method includes the following steps: an image extraction device of the object detection device extracts a plurality of images of a specific space and a plurality of depth images, each of which corresponds to one of the depth images; a processor of the object detection device then uses an object detection model to detect objects in each image and obtain object information; the processor then calculates a depth value of each object in each depth image; the processor then inputs the depth value of each object into an object state model to determine the object state of each object; and the processor then determines whether to enter the specific space based on the object information and the object state of each object.

[0008] With respect to other aspects and features of the present invention, those skilled in the art can make various modifications based on the object detection device and method according to the embodiments of the present invention without departing from the spirit of the present invention. [Effects of the Invention]

[0009] The present invention can assist the AMR in determining the appropriate time to enter the elevator. [Brief explanation of the drawings]

[0010] The present invention may be more fully understood by reference to the following detailed description and examples taken in conjunction with the accompanying drawings, in which: [Figure 1] 1 is a block diagram of an object detection device 100 according to an embodiment of the present invention. [Figure 2A] FIG. 2 illustrates an image according to one embodiment of the present invention. [Figure 2B] FIG. 2 illustrates a depth image according to one embodiment of the present invention. [Figure 3] 1 is a flowchart illustrating an object detection method according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating step S350 according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following description sets forth the best mode for carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best defined by reference to the appended claims.

[0012] FIG. 1 is a block diagram of an object detection device 100 according to an embodiment of the present invention. The object detection device 100 is an autonomous mobile transport robot (AMR). As shown in FIG. 1, the object detection device 100 includes an image extraction device 110, an infrared detection device 120, a storage device 130, a communication device 140, and a processor 150. Note that FIG. 1 is a simplified block diagram showing only elements relevant to the present invention, and the present invention is not limited to the contents of FIG. 1. The object detection device 100 also includes other elements, such as a drive unit, a transmission mechanism, a motor, or wheels, that drive the movement of the AMR. The drive unit moves the AMR forward or backward based on instructions from the processor 150.

[0013] According to an embodiment of the present invention, the image extraction device 110 is a camera or other electronic device with a photographing function. The image extraction device 110 extracts images of a specific space (e.g., inside an elevator) for a certain period of time (e.g., 1 second) to generate a certain number of images and depth images. For example, when the elevator door opens, the image extraction device 110 captures the interior of the elevator and generates 30 consecutive images and 30 consecutive depth images every second (i.e., a certain period of time). It should be noted that the present invention should not be limited by the above example. In several embodiments of the present invention, each image corresponds to one depth image. Taking FIGS. 2A and 2B as an example, the image in FIG. 2A corresponds to the depth image in FIG. 2B.

[0014] According to an embodiment of the present invention, infrared detection device 120 uses infrared rays to detect whether there is any obstacle around object detection device 100. For example, in response to infrared detection device 120 detecting an obstacle around object detection device 100, or in response to infrared detection device 120 detecting an object approaching object detection device 100, processor 150 instructs object detection device 100 to stop movement and generate a warning sound or a warning signal.

[0015] According to an embodiment of the present invention, storage device 130 stores software and firmware program code, system data, user data, etc. for object detection apparatus 100. Storage device 130 may be volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., flash memory, read-only memory (ROM)), a hard disk, or a combination of the above memory devices.

[0016] According to an embodiment of the present invention, the communication device 140 communicates with a control device of a specific space (e.g., an elevator). The communication device 140 receives a wireless signal (e.g., a Bluetooth signal or a Wi-Fi signal, but the present invention should not be limited thereto) from the control device of the specific space to determine whether the elevator door is open or closed. In response to the signal indicating that the elevator door is open, the object detection device 100 performs an object detection operation to determine whether to enter the elevator. In response to the signal indicating that the elevator door is closed, the object detection device 100 stops performing the object detection operation and waits for the next signal from the elevator indicating that the elevator door is open.

[0017] According to an embodiment of the present invention, the processor 150 controls the operation of the image extraction device 110, the infrared detection device 120, the storage device 130, and the communication device 140. According to an embodiment of the present invention, the processor 150 is further configured to execute program code of a software module to perform an object detection operation. When the program code is executed, the program code along with specific data in a data structure is also referred to as a processor logic unit or stack instance. Thus, the processor 150 can be considered to have multiple processor logic units, each of which performs one or more specific functions or tasks of a corresponding software module. According to an embodiment of the present invention, the processor is a central processing unit (CPU), a controller, or a control chip, although the present invention should not be limited thereto.

[0018] According to one embodiment of the present invention, the object detection device 100 obtains a certain number (e.g., 30) of images and depth images of a specific space (e.g., in an elevator) over a certain period (e.g., 1 second). Then, the object detection device 100 uses an object detection model to detect objects in each image and obtain object information. The depth images further include depth information corresponding to the object information. The object information includes the type of object included in each image (e.g., a person, wheelchair, stroller, cart, etc.). The object information includes the location of each object in the image (i.e., its location in the specific space). According to one embodiment of the present invention, the object detection model is a You Only Look Once (YOLO) model or a Region Convolutional Neural Network (R-CNN) model, but the present invention should not be limited thereto. In addition, the object detection device 100 tracks the same object in each image based on an object tracking algorithm to confirm that the object detected in each image generated over a certain period (e.g., 1 second) is the same object. According to one embodiment of the present invention, based on an object tracking algorithm, the object detection device 100 calculates the distance between an object in a first image and an object in a second image, and then the object detection device 100 considers the object in the first image and the object in the second image that have the shortest distance to be the same object.

[0019] According to one embodiment of the present invention, the object detection device 100 calculates a depth value corresponding to each object in the depth image. In particular, the object detection device 100 calculates a depth matrix for each object in the depth image. Taking FIGS. 2A and 2B as examples, when the object detection device 100 detects object O1, the object detection device 100 frames object O1 using frame M1. The depth values corresponding to frame M1 are the depth matrix corresponding to object O1. The object detection device 100 then arranges the depth values in the depth matrix in order, selects a specific range of depth values from the arranged depth values in the depth matrix (e.g., selects a specific percentage of depth values from the middle part of the arranged depth values), and calculates the average value of the selected depth values as the depth value corresponding to the object. In this way, the object detection device 100 obtains the depth value of each object in the depth image. For example, if the depth matrix corresponding to an object in the depth image is a 5x5 matrix:

[0020] TIFF0007720362000001.tif32170

[0021] After the object detection device 100 sorts the depth values in the depth matrix, the depth values in the depth matrix change to {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13...25}. Then, the object detection device 100 selects depth values (i.e., a specific range) that are 30% to 60% of the sorted depth values (i.e., selects {8, 9, 10, 11, 12, 13, 14, 15}) and averages the selected depth values (i.e., (8 + 9 + 10 + 11 + 12 + 13 + 14 + 15) / 8 = 11.5) to obtain a depth value corresponding to the object (i.e., 11.5). It should be noted that the above examples are used only to explain embodiments of the present invention, and the present invention should not be limited thereto. Because a frame corresponding to an object is surrounded by certain background noise, the background noise in the frame is filtered by the above scheme to obtain the depth value of the object.

[0022] According to an embodiment of the present invention, after the object detection device 100 obtains the depth values of each object over a certain period of time, the object detection device 100 inputs the depth values of each object into an object state model to determine the object state of each object. According to an embodiment of the present invention, the object state includes an object leaving a specific space (e.g., an elevator) or an object remaining in a specific space (e.g., an elevator). In addition, according to an embodiment of the present invention, the object state model may be, for example, a recurrent neural network (RNN) model or a long short-term memory (LSTM) model, but the present invention should not be limited thereto. For example, if the depth values of the objects over a certain period of time are {20, 20, 19, 18, 17, 17, 15, 13, 13, 12}, when the object detection device 100 inputs the object depth values into the LSTM model, the object detection device 100 will know that the object is leaving the specific space (e.g., an elevator). It should be noted that the above examples are used only to describe embodiments of the present invention, and the present invention should not be limited thereto.

[0023] According to one embodiment of the present invention, the object detection device 100 determines whether to enter a specific space (e.g., an elevator) based on the object information and the object state of each object. In particular, the object detection device 100 first determines whether the object includes at least one specific object (e.g., a wheelchair, a stroller, or a cart) based on the object information.

[0024] In response to the object having at least one specific object, the object detection device 100 retreats a certain distance to provide sufficient space for the specific object (e.g., a wheelchair, a stroller, or a cart) to leave the specific space (e.g., an elevator). Thereafter, the object detection device 100 continuously monitors whether the specific object has already left the specific space (i.e., the object detection device 100 continuously monitors the object status of the specific object at regular intervals). In response to the specific object not having left the specific space, the object detection device 100 determines not to enter the specific space. In response to the specific object having already left the specific space, the object detection device 100 further determines whether the number of objects (e.g., the number of people) in the specific space exceeds a threshold (e.g., five people, but the present invention should not be limited thereto). In response to the number of objects in the specific space exceeding the threshold, the object detection device 100 further determines not to enter the specific space. In response to the fact that the number of objects in the specific space does not exceed the threshold, object detection device 100 determines that the object is entering the specific space.

[0025] In response to the object not having any specific object, the object detection device 100 further determines whether the quantity of objects (e.g., the number of people) in the specific space exceeds a threshold (e.g., five people, but the present invention should not be limited thereto). In response to the quantity of objects in the specific space exceeding the threshold, the object detection device 100 further determines not to enter the specific space. In response to the quantity of objects in the specific space not exceeding the threshold, the object detection device 100 determines to enter the specific space.

[0026] In response to the object detection device 100 determining that there is no object in the specific space based on the object information, the object detection device 100 determines to enter the specific space, or the object detection device 100 first waits for a certain period of time and then enters the specific space. In response to the specific space being closed (e.g., the elevator doors being closed), the object detection device 100 stops performing the object detection operation and stays and waits at its original position until the specific space is opened again (e.g., the elevator doors are opened again).

[0027] 3 is a flowchart illustrating an object detection method according to an embodiment of the present invention. The procedure of FIG. 3 is applied to the object detection apparatus 100. As shown in FIG. 3, in step S310, the object detection apparatus 100 extracts a plurality of images of a specific space and a plurality of depth images, each of which corresponds to one of the depth images.

[0028] In step S320, the object detection apparatus 100 detects an object in each image using an object detection model to obtain object information. According to one embodiment of the present invention, the object detection model is a YOLO (You Only Look Once) model or a regional convolutional neural network (R-CNN) model, but the present invention should not be limited thereto.

[0029] In step S330, object detection apparatus 100 calculates the depth value of each object in each depth image.

[0030] In step S340, the object detection apparatus 100 inputs the depth value of each object into an object state model to determine the object state of each object. According to one embodiment of the present invention, the object state model is a recurrent neural network (RNN) model or a long short-term memory (LSTM) model, but the present invention should not be limited thereto.

[0031] In step S350, object detection device 100 determines whether to enter the specific space based on the object information and the object state of each object. In one embodiment, object detection device 100 repeats steps S310 to S330 at regular intervals to track each object. Thereafter, object detection device 100 executes step S340 to determine the object state of each object, and also executes step S350 to determine whether to enter the specific space based on the object information and the object state of each object.

[0032] Fig. 4 is a flowchart illustrating step S350 according to an embodiment of the present invention. The procedure in Fig. 4 can be applied to the object detection device 100. As shown in Fig. 4, in step S410, the object detection device 100 determines whether the object has at least one specific object based on the object information.

[0033] In response to the object not having any particular object, step S420 is executed. In step S420, object detection apparatus 100 determines whether the quantity of objects exceeds a threshold value.

[0034] In response to the fact that the number of objects does not exceed the threshold, step S430 is executed. In step S430, object detection device 100 determines that the object is entering a specific space.

[0035] In response to the quantity of objects exceeding the threshold, step S440 is executed. In step S440, object detection device 100 determines not to enter the specific space. After step S440, object detection device 100 performs the operations of FIG. 4 again at the next fixed period. In addition, after step S440, object detection device 100 also performs the operations of FIGS. 3 and 4 again after receiving the next signal indicating that the specific space (e.g., an elevator door) has opened.

[0036] In response to the object having the specific object, step S450 is executed, in which object detection apparatus 100 retreats a certain distance (e.g., 1 meter) so that the specific object (e.g., a wheelchair, a stroller, or a cart) has enough space to leave the specific space (e.g., an elevator).

[0037] In step S460, the object detection device 100 determines whether the specific object wants to leave the specific space based on the object state of the specific object. In response to the object detection device 100 determining that the specific object wants to leave the specific space, the object detection device 100 executes step S410 and resumes the operation of FIG. 4 to continuously determine whether the specific object has already left the specific space. In response to the object detection device 100 determining that the specific object does not want to leave the specific space, the object detection device 100 determines not to enter the specific space (i.e., executes step S440).

[0038] According to the object detection method provided in the present invention, the object detection device can determine the appropriate time to enter a specific space (e.g., an elevator).

[0039] In this disclosure and in the claims, ordinal numbers such as "first," "second," and "third" are used for convenience of description and do not have any sequential relationship to one another.

[0040] The method and algorithm steps disclosed herein may be implemented directly in a processor, in hardware, in software modules, or in a combination of the two. Software modules (e.g., having executable instructions and associated data) and other data may reside in data memory, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable storage medium known in the art. The storage medium may be coupled to, for example, a computer / processor (referred to herein as the "processor" for brevity) such that the processor can read information (e.g., code) from, or write information to, the storage medium. The storage medium may be integrated with the processor. An ASIC may comprise a processor and a storage medium. A user device may comprise an ASIC. That is, the processor and storage medium may be included in the user device in a manner not directly connected to the user device. Furthermore, in some embodiments, any suitable computer-program product may comprise a computer-readable medium containing code related to one or more aspects of the present disclosure. In some aspects, a computer program product comprises packaging materials.

[0041] The above paragraphs describe various embodiments. Clearly, the teachings of the present invention can be implemented in a variety of ways, and any specific structure or function disclosed is merely representative. Given the teachings herein, one skilled in the art can apply them singly or in combination with two or more other embodiments.

[0042] Although preferred embodiments of the present invention have been disclosed as described above, these are by no means intended to limit the scope of the present invention, and anyone familiar with the art can make various modifications within the scope of the present invention. [Explanation of symbols]

[0043] 110 Image Extraction Device 120 Infrared detector 130 Storage Devices 140 Communication equipment 150 processors M1 Frame 01 Object S310~S350, S410~S460 process

Claims

1. An object detection device, an image extraction device for extracting a plurality of images of a specific space and a plurality of depth images, each image corresponding to one of the plurality of depth images; a processor coupled to the image extraction device for obtaining the plurality of images and the plurality of depth images from the image extraction device; and The processor detects an object in each image using an object detection model to obtain object information; the processor calculates a depth value for each object in each depth image; the processor inputs the depth values for each object into an object state model to determine an object state for each object; the processor determines whether the object detection device will enter the specific space based on the object information and the object state of each object; The processor determines whether the object has at least one specific object based on the object information, and in response to the object having the at least one specific object, the processor instructs the object detection device to retreat, the processor determines whether to enter the specific space according to the object state of the at least one specific object, and in response to the object not having the at least one specific object, the processor determines that the quantity of the objects exceeds a threshold, and in response to the quantity of the objects exceeding the threshold, the processor instructs the object detection device not to enter the specific space, and in response to the quantity of the objects not exceeding the threshold, the processor instructs the object detection device to enter the specific space, the specific object being a wheelchair, a stroller, or a cart. An object detection device characterized by:

2. The object detection device of claim 1, characterized in that the processor tracks the same object in each image, calculates a depth matrix for each object in each depth image, arranges the depth values of the depth matrix in order, selects a specific range of the depth values from the order, and calculates an average value of the selected depth values to obtain the depth value of each object in each depth image.

3. An object detection method applied to an object detection device, extracting, with an image extraction device, a plurality of images of the particular space and a plurality of depth images, each image corresponding to one of the plurality of depth images; detecting, by a processor, an object in each image using an object detection model to obtain object information; calculating, by the processor, a depth value for each object in each depth image; inputting, by the processor, the depth values for each object into an object state model to determine an object state for each object; determining, by the processor, whether the object detection device will enter the specific space based on the object information and the object state of each object; and The object detection method further comprises: determining, by the processor, based on the object information, whether the object has at least one particular object; instructing, by the processor, the object detection device to back up in response to the object having the at least one particular object; and The method further comprises: determining, by the processor, whether to enter the specific space according to the object state of the at least one specific object; determining, by the processor, in response to the object not having the at least one particular object, that a quantity of the object exceeds a threshold; In response to the quantity of the objects exceeding a threshold, instructing the object detection device not to enter the specific space by the processor; instructing the object detection device to enter the specific space by the processor in response to the quantity of the objects not exceeding the threshold; wherein the specific object is a wheelchair, a stroller, or a cart.

Images