Object recognition model training device, object recognition device, and object recognition model training method

Abstract

Provided are a device and method capable of accurately recognizing an arbitrary object from an image.　A training information acquisition unit 101 acquires RGB data of a captured image, depth data for a to-be-imaged object (captured in the captured image such as a stump K) included in the captured image, and annotation data indicating the position of the stump K (to-be-recognized object) included in the captured image. A training unit 103 trains an object recognition model 201 on the basis of the RGB data, the depth data, and the annotation data.

Description

Object recognition model learning device, object recognition device, and object recognition model learning method

[0001] The present invention relates to an object recognition model learning device, an object recognition device, and an object recognition model learning method for performing image recognition.

[0002] Patent document 1 describes generating an inference model that learns information about a training image and an object from training images and teacher data about the object depicted in the training image, and then using this inference model to infer the object.

[0003] Japanese Patent Application Laid-Open No. 2023-1747

[0004] However, there are some objects that are difficult to identify using only an image (e.g., only an RGB image). For example, when trying to recognize a stump in a forest from an aerial photograph, accurate recognition may not be possible using only an RGB image.

[0005] Therefore, an object of the present disclosure is to provide an object recognition model learning device, an object recognition device, and an object recognition model learning method that can accurately recognize any object from an image.

[0006] The object recognition model learning device of the present disclosure includes a learning information acquisition unit that acquires a captured image, depth information for a captured object included in the captured image, and annotation information indicating the position of a recognition object included in the captured image, and a learning unit that learns an object recognition model based on the captured image, depth information, and annotation information.

[0007] In addition, the object recognition device of the present disclosure includes an object detection model trained based on a captured image, depth information for a subject included in the captured image, and annotation information indicating the position of a detection target included in the captured image, an image acquisition unit that acquires the captured image, and a recognition unit that recognizes the detection target using the object detection model based on the captured image.

[0008] According to the present disclosure, it is possible to learn an object recognition model that can perform accurate object recognition.

[0009] FIG. 1 is a diagram illustrating an overview of the processing of the present disclosure. FIG. 2 is a diagram illustrating the functional configuration of a learning device 100 that learns an object recognition model for inferring a stump from a captured image of the present disclosure. FIG. 3 is a diagram illustrating RGB data for training. FIG. 4 is a diagram illustrating details of a process for generating RGBD data. FIG. 5 is a flowchart illustrating the learning processing of the learning device 100. FIG. 6 is a schematic diagram illustrating the learning processing of the learning device 100. FIG. 7 is a diagram illustrating the functional configuration of an object recognition device 200. FIG. 8 is a flowchart illustrating the operation of the object recognition device 200. FIG. 9 is a diagram illustrating recognition processing based on RGB data and RGBD data. FIG. 10 is a diagram (part 1) illustrating the results of recognition by the object recognition device 200. FIG. 11 is a diagram (part 2) illustrating the results of recognition by the object recognition device 200. FIG. 12 is a diagram illustrating an example of the hardware configuration of the learning device 100 and the object recognition device 200 according to an embodiment of the present disclosure.

[0010] The present disclosure will be described with reference to the accompanying drawings. Whenever possible, the same parts are designated by the same reference numerals and redundant description will be omitted.

[0011] 1A and 1B are diagrams illustrating an overview of the processing of the present disclosure. As shown in Fig. 1A, the present disclosure relates to processing of aerial photographs (hereinafter referred to as photographed images) taken by a drone D. Naturally, the present disclosure is not limited to drone D, and may be applied to photography using flying objects such as airplanes, helicopters, and balloons, as well as photography using other means capable of taking images from high positions.

[0012] FIG. 1( a) shows a drone D photographing a forest from above. The forest includes a stump K. In FIG. 1( a), the ground G is shown, and the stump K protrudes slightly from the ground G. The camera of the drone D has a depth camera function that can also acquire depth data, and can acquire depth data in addition to the RGB data of the captured image. As described below, the depth data can also be acquired from the RGB data. In this case, the camera of the drone D acquires only the RGB data, and the learning device 100 or object recognition device 200, described below, acquires the depth data from the RGB data. The depth data indicates the distance from the camera.

[0013] 1B is a diagram showing that the image captured by the drone D is stored in the image data storage unit 250. The captured image is treated as RGB data for learning to train an object recognition model for recognizing the stump K or as RGB data as an inference target.

[0014] 2 is a diagram showing the functional configuration of a learning device 100 according to the present disclosure that learns an object recognition model for inferring a stump K from a captured image. As shown in the figure, the learning device 100 includes a learning information acquisition unit 101, a generation unit 102, and a learning unit 103. The learning device 100 is configured to be able to access an image data storage unit 250 and an object recognition model storage unit 260, retrieves RGB data from the image data storage unit 250, and stores the learned object recognition model in the object recognition model storage unit 260. The image data storage unit 250 and the object recognition model storage unit 260 may be included in the learning device 100.

[0015] The learning information acquisition unit 101 is a part that acquires learning RGB data and depth data as learning information from the image data storage unit 250. This learning information includes annotation data assigned by the administrator of the RGB data in addition to the RGB data and depth data acquired by the drone D. The learning information will now be described. FIG. 3 is a diagram showing learning RGB data. As shown in the figure, the RGB data S includes a stump K and annotation data an indicating the stump K. In FIG. 3, annotation data an is assigned to each stump K. For example, annotation data an1 is assigned to stump K1. This annotation data an is assigned by the administrator of the RGB data. In the present disclosure, annotation data an is assigned in the form of a rectangle surrounding the stump K, but the shape, etc., is not limited to this.

[0016] The generation unit 102 generates RGBD data for training (synthetic captured image) based on RGB data and depth data, which are training information. More specifically, the generation unit 102 first divides the RGB data into arbitrary grid sizes to obtain multiple pieces of divided RGB data. Furthermore, based on the depth data, the generation unit 102 generates grayscale data that is normalized from 0 to 255 and grayscaled for each arbitrary grid size, similar to the RGB data. The generation unit 102 divides the grayscale data into arbitrary grid sizes, superimposes it on RGB data of the same grid size, and changes the brightness according to the grayscale of the grayscale data to generate RGB data + depth data (RGBD data). The generation unit 102 normalizes the brightness of the RGBD data. In the present disclosure, the depth data is expressed in terms of brightness, etc., and the information content of the depth data is embedded in the RGB data.

[0017] FIG. 4 is a diagram showing details of the RGBD data generation process. FIG. 4(a) shows RGB data of a captured image. FIG. 4(b) shows grayscale data based on depth data. The grayscale data is expressed in gradations of black (or gray) depending on the elevation difference, such as the part corresponding to the stump K. Because the stump K is formed slightly higher than the ground (the distance indicated by the depth data is short), the depth data also reflects this (black). FIG. 4(c) is a diagram showing grayscale superimposed on RGB data. In the present disclosure, superimposition refers to adjusting the brightness of the RGB data according to the depth data indicated by the grayscale data. The adjustment here refers to a process of reducing the brightness according to the depth data. Details will be described later. In FIG. 4(b), parts other than the stump K are also expressed in black. This is because parts other than the stump K also have elevation differences.

[0018] In this way, the generating unit 102 generates RGB data and RGBD data on which grayscale data is superimposed.

[0019] The learning unit 103 is a part that learns an object recognition model for recognizing the stump K based on the generated RGB data and RGBD data and annotation data. That is, the portions of the RGB data and RGBD data (input data) indicated by the annotation data are the targets of object recognition (teaching data), and the learning unit 103 learns to recognize them from the RGB data and RGBD data. The annotation data is provided in the RGB data, and may be reflected in the RGBD data or may be provided in each of the RGB data.

[0020] In this disclosure, YOLOv8 is used as the object recognition model for training. YOLOv8 is an object recognition model whose greatest feature is that it directly predicts the center point of an object in RGB data and infers the dimensions of the object's bounding box from that center. Naturally, object recognition models other than YOLOv8 may also be applied, such as R-CNN and SSD.

[0021] The object recognition model storage unit 260 stores the object recognition model learned by the learning unit 103. The object recognition model stored here is used to infer the stump K in the captured image.

[0022] 5 is a flowchart showing the learning process of the learning device 100. The learning information acquisition unit 101 acquires learning information (RGB data, depth data, annotation data) from the image data storage unit 250 (S101). The generation unit 102 divides the RGB data into grids of an arbitrary size (S102).

[0023] The generating unit 102 grayscales the depth data at an arbitrary grid size (S103). The generating unit 102 superimposes the grayscale data (depth data) grayscaled at an arbitrary grid size on the RGB data divided at the arbitrary grid size to generate RGBD data with changed brightness (S104).

[0024] The learning unit 103 performs learning based on the RGB data, the RGBD data, and the respective annotation data, and generates an object recognition model (for example, YOLOv8) (S105). The learning unit 103 stores the generated object recognition model in the object recognition model storage unit 260.

[0025] 6 is a schematic diagram illustrating this learning process. As described above, the learning information acquisition unit 101 acquires RGB data and depth data (corresponding to S101). In the present disclosure, the depth data is obtained using a DSM (Digital Surface Model).

[0026] The generation unit 102 generates RGB data by dividing the RGB data by an arbitrary grid size (corresponding to S102). Meanwhile, the generation unit 102 normalizes the depth data by the arbitrary grid size and converts it to grayscale to generate grayscale data (corresponding to process S103). Here, the generation unit 102 superimposes the RGB data and the depth data (corresponding to process S104). Here, since the grid size is the same, the RGB data and the depth data are the same size. A learning process is performed using the RGB data, RGBD data, and annotation data (description of which is omitted here) generated in this manner.

[0027] Next, a process of recognizing a stump K from a captured image using the learned object recognition model 201 will be described. FIG. 7 is a diagram showing the functional configuration of an object recognition device 200 equipped with the object recognition model 201. As shown in the diagram, the object recognition device 200 includes the object recognition model 201, a captured image acquisition unit 202, a composite captured image acquisition unit 203, and a recognition unit 204.

[0028] As described above, the object recognition model 201 is an inference model for recognizing objects learned by YOLOv8, and is an inference model learned by the learning device 100.

[0029] The captured image acquisition unit 202 is a part that acquires the captured image (RGB data) stored in the image data storage unit 250. As described above, the image data storage unit 250 stores the captured image captured by the drone D.

[0030] The composite captured image acquisition unit 203 generates RGBD data from the RGB data and depth data and acquires the generated RGBD data as a composite captured image. The generation of the RGBD data is as described above.

[0031] The recognition unit 204 is a part that inputs at least one of the RGB data acquired by the captured image acquisition unit 202 and the RGBD data acquired by the composite captured image acquisition unit 203 into the object recognition model 201 and obtains an output result, thereby recognizing the stump K in the RGB data. The recognition unit 204 stores the recognition result in the recognition result storage unit 270. In the present disclosure, the recognition unit 204 generates data that has been processed from the RGB data by circling the position of the stump K in red.

[0032] The recognition result storage unit 270 is a part that stores the position of the stump K in the RGB data recognized by the recognition unit 204. In the present disclosure, the RGB data is processed by circumscribing the position of the stump K in red, and data is stored.

[0033] The processing contents of the object recognition device 200 configured as above will be described. Fig. 8 is a flowchart showing the operation of the object recognition device 200. As shown in the figure, the captured image acquisition unit 202 acquires the captured image (RGB data) and depth data from the image data storage unit 250 (S201). The recognition unit 204 divides the RGB data into an arbitrary grid size (S202).

[0034] Meanwhile, the composite captured image acquisition unit 203 grayscales the depth data at an arbitrary grid size to generate grayscale data (S203). The composite captured image acquisition unit 203 superimposes the grayscale data grayscaled at an arbitrary grid size on the RGB data divided at the arbitrary grid size to generate RGBD data with changed brightness (S204). This RGBD data corresponds to the composite captured image.

[0035] The recognition unit 204 inputs the RGBD data into the object recognition model 201 and obtains a recognition result, thereby performing object recognition (S205). In the present disclosure, the recognition unit 204 obtains a captured image showing the position of the stump.

[0036] In the above process, the object recognition device 200 generates RGBD data and inputs it to the object recognition model, but this is not limited to this. Only RGB data may be input to the object recognition model 201 to obtain the results. When only RGB data is used, steps S203 and S204 can be omitted.

[0037] 9A and 9B are diagrams showing the recognition process based on RGB data and RGBD data. Fig. 9A is a schematic diagram showing the recognition process using RGB data. As shown in the figure, the acquired RGB data (corresponding to S201) is divided into an arbitrary grid size (corresponding to process S202), which is input to an object recognition model 201, where object recognition is performed (corresponding to process S205).

[0038] 9B is a diagram showing the recognition process using RGB data. As shown in the figure, first, RGB data representing the captured image is obtained (corresponding to process S201). Meanwhile, the depth data is grayscaled at an arbitrary grid size (corresponding to process S203), and the depth data and RGB data are superimposed to generate RGBD data (corresponding to process S204). The RGBD data is input into an object recognition model 201, where object recognition is performed (corresponding to process S205).

[0039] In the present disclosure, as shown in Fig. 9(a), object recognition can be performed in the same way as with existing methods if only RGB data is available, and the recognition accuracy can be improved by adding depth data. For example, a use case is possible in which primary screening is performed by performing object recognition using RGB data captured by a drone as is, and then more detailed analysis is performed by adding depth data using photogrammetry or the like.

[0040] Next, a detailed description will be given of the process of superimposing RGB data and depth data in the generation unit 102 of the learning device 100 and the composite captured image acquisition unit 203 of the object recognition device 200 according to the present disclosure. The following description will be given using the generation unit 102, but the composite captured image acquisition unit 203 also performs similar processing.

[0041] As described above, the generation unit 102 first obtains depth data (grayscale data) that is normalized to 8 bits for each arbitrary grid size and converted to grayscale data. This grid size is the same as the size of the image that is divided when input to the object recognition model 201. The normalization of the depth data is performed based on the following equation (1). (x, y) are the coordinates of the grid, (z, w) are the coordinates of a pixel within the grid, Pnorm is the normalized pixel within the grid, DSM is the depth data value, Pmin is the minimum value within the grid, and Pmax is the maximum value within the grid.

[0042] Next, the generation unit 102 superimposes the grayscale image onto the RGB data for each grid and changes the brightness. In the brightness processing step, the RGB data is converted into the HSV color space and then the brightness value is reduced. The brightness is changed based on Equation 2. Here, Vconv is the brightness after superimposition, V is the brightness before superimposition, and MaxV is the maximum amount of brightness reduction. MaxV is a value set in advance. In the present disclosure, brightness is reduced, but this is not limiting and brightness may be varied. For example, brightness may be increased, or the brightness may be increased or decreased depending on the overall brightness of the captured image (RGB data) to highlight the area around the stump K.

[0043] Furthermore, in the above description, the process involves changing the lightness, but this is not limiting. Instead of or in addition to changing the lightness, the hue or saturation in the HSV space may be changed (increased or decreased).

[0044] In addition to or apart from the above, although the color may appear strange to the human eye, changes in the RGB color space (for example, emphasizing or suppressing only R) will produce changes in the data, and have the effect of highlighting objects such as the stump K. In the present disclosure, change methods that are not limited to color spaces can be applied.

[0045] For example, color spaces available in openCV include RGB, HSV, HLS, Lab, LuvXYZ, and YUV, which can be applied to change the object to make it stand out. That is, you can convert to another color space and change certain values ​​to achieve the same effect.

[0046] Next, we will provide additional explanation regarding the division of RGB data into an arbitrary grid size. When inputting RGB data or RGBD data into the object recognition model 201, it is preferable to divide the data during the processing process to reduce the influence of mountain slopes. This is because normalization over the entire field would result in the characteristics of the stump to be recognized being lost due to the influence of ground height. In addition, here, we assume that the learning model will capture the characteristics of the stump K and divide the data into a size that can be learned; for example, the size of the stump K is used as the grid size.

[0047] In the above description of this disclosure, a stump K is used as the recognition target, but the present invention can be applied to all objects on the ground. For example, stones, rocks, building foundations and foundation piles, road sign posts, fence posts, pallets and boxes placed on the ground, collapsed earth and rubble, etc. can be recognized as recognition targets. The present invention can also be applied to the recognition of victims and fallen airplanes.

[0048] As mentioned above, this technology can be applied to any image or video that has RGB data, including aerial photographs and other photographs and images of non-terrestrial objects, as long as the RGB data is available. For example, even if the RGB data is poorly captured (e.g., it is dim and the previous 3ch (RGB data only, without depth data) had poor inference accuracy), the superimposition of depth data clarifies the contours and background, improving the accuracy of object recognition.

[0049] Next, the effects of the learning device 100 and object recognition device 200 of the present disclosure will be described. In the learning device 100 of the present disclosure, the learning information acquisition unit 101 acquires RGB data of a captured image, depth data for a captured object (such as a stump K or something else that appears in the captured image) included in the captured image, and annotation data indicating the position of the stump K (object to be recognized) included in the captured image. The learning unit 103 then learns the object recognition model 201 based on the RGB data, the depth data, and the annotation data.

[0050] In the present disclosure, the generation unit 102 generates RGBD data (synthesized captured image) by combining RGB data with depth data. The learning unit 103 then learns the object recognition model 201 based on the RGB data, RGBD data, and annotation data. In the above disclosure, both RGB data and RGBD data are used, but it is preferable to use at least RGBD data.

[0051] The generation unit 102 then generates RGBD data (a composite captured image) by changing the image configuration information other than the part (specific part) with a difference in elevation that includes the stump K indicated by the depth data. The image configuration information in the present disclosure includes color space information.

[0052] Furthermore, the learning information acquisition unit 101 divides the captured image into grids of an arbitrary size, and generates grayscale data of the arbitrary grid size based on the depth data.

[0053] According to this configuration, the object recognition model 201 can be learned from the captured image (RGB data), depth data, and annotation data, and can be used to enable accurate object recognition.

[0054] Moreover, the object recognition device 200 of the present disclosure includes an object recognition model 201 that is trained based on RGB data, depth data for a captured object (an object captured in a captured image, such as a stump K) included in the RGB data, and annotation data indicating the position of the stump K (recognition target) included in the RGB data. The captured image acquisition unit 202 acquires the RGB data, and the recognition unit 204 recognizes the stump K (recognition target) using the object recognition model 201 based on the RGB data. For example, the object recognition model 201 may input only RGB data or RGBD data.

[0055] This allows the object to be accurately recognized from the RGB data.

[0056] When RGBD data is input, the following configuration is further adopted. That is, in the present disclosure, object recognition device 200 further includes composite captured image acquisition unit 203 that acquires RGBD data (composite captured image) that includes depth data for the captured object contained in the RGB data. Recognition unit 204 inputs the RGBD data to object recognition model 201 and recognizes the recognition target.

[0057] This allows for more accurate object recognition.

[0058] In addition, in the present disclosure, the recognition unit 204 inputs the RGB data into the object recognition model 210 to obtain a recognition result. Then, the recognition unit 204 recognizes the recognition target based on a captured image based on this recognition result and the RGBD data obtained from the captured image. That is, screening is performed using only the RGB data to narrow down the target RGB data, and then object recognition is performed using the RGBD data. This allows for highly accurate recognition.

[0059] In the present disclosure, the object recognition model 201 is trained by the training device 100 .

[0060] The effects of the present disclosure will be described with reference to the drawings. Figures 10 and 11 show the results of recognition by the object recognition device 200. Figure 10 shows the comparison results when RGB data and RGBD data are input, respectively.

[0061] FIG. 10(a) shows RGB data, which is the data to be processed. The boxed area indicates a stump K, which is the recognized portion. FIG. 10(b) shows the results of object recognition when RGB data is input to an object recognition model trained only on RGB data. Here, the ground and the fallen tree (indicated by the symbol x) are misrecognized. FIG. 10(c) shows the results of inferring RGB data using an object recognition model 201 trained on RGB data and RGBD data (brightness V reduced by a maximum of -25). FIG. 10(d) shows the results of inferring RGB data using an object recognition model 201 trained on RGB data and RGBD data (brightness V reduced by a maximum of -25). As shown in the figure, FIG. 10(b) shows a misrecognition, while FIG. 10(c) and FIG. 10(d) show the same results as FIG. 10(a).

[0062] This shows that learning the object recognition model 201 using RGBD data makes it possible to learn an object recognition model 201 with high accuracy.

[0063] FIG. 11 shows the results of using an object recognition model 201 trained by varying the maximum reduction in brightness V. FIG. 11(a) shows RGB data, which is the data to be processed. The boxed areas x1 to x3 indicate the stump K. FIG. 11(b) shows grayscale data obtained by normalizing the depth data and converting it to grayscale. FIG. 11(c) shows the results of inferring RGB data using a model trained on RGB data and RGBD data (brightness V reduced by a maximum of -25). Here, the object marked x4, which has a shape similar to the stump K, is incorrectly recognized. Meanwhile, the two stumps K (the portions corresponding to marks x2 and x3) are not recognized. FIG. 11(d) shows the results of inferring RGB data using an object recognition model 201 trained on RGB data and RGBD data (brightness V reduced by a maximum of -50). In this result, the portion corresponding to x3, which corresponds to the stump in the lower left corner and has a small cross-section, is not recognized. From this result, it can be seen that by increasing the amount of change in brightness V, accurate inference becomes possible.

[0064] The device and method of the present disclosure have the following configuration.

[0065] [1] An object recognition model learning device comprising: a learning information acquisition unit that acquires a captured image, depth information for a captured object included in the captured image, and annotation information indicating the position of a recognition target included in the captured image; and a learning unit that learns an object recognition model based on the captured image, the depth information, and the annotation information.

[0066] [2] The object recognition model learning device according to [1], further comprising a generation unit that generates a composite captured image by combining the captured image with the depth information, wherein the learning unit learns an object recognition model based on the captured image, the composite captured image, and the annotation information.

[0067] [3] The object recognition model learning device according to [2], wherein the generation unit generates a synthetic captured image by changing image configuration information that configures the captured image according to a specific portion indicated by the depth information.

[0068] [4] The object recognition model learning device according to [2] or [3], wherein the learning unit performs learning based on the captured image, the composite captured image, and an annotation image.

[0069] [5] The object recognition model learning device according to any one of [1] to [4], wherein the learning information acquisition unit divides the captured image into an arbitrary grid size and generates a depth image of the arbitrary grid size based on the depth information.

[0070] [6] An object recognition device comprising: an object recognition model trained based on a captured image, depth information for a subject included in the captured image, and annotation information indicating the position of a recognition target included in the captured image; a captured image acquisition unit that acquires the captured image; and a recognition unit that recognizes the recognition target using the object recognition model based on the captured image.

[0071] [7] The object recognition device according to [6], further comprising a composite captured image acquisition unit that acquires a composite captured image including depth information for a target object included in the captured image, and the recognition unit inputs the captured image and the composite captured image into the object recognition model to recognize the target object.

[0072] [8] The object recognition device according to [7], wherein the recognition unit inputs the captured image into the object recognition model to obtain a recognition result, and recognizes the recognition target based on a captured image based on the recognition result and a composite captured image obtained from the captured image.

[0073] [9] The object recognition device according to any one of [6] to [8], wherein the object recognition model is learned by the object recognition model learning device according to any one of [1] to [4].

[0074]

[10] An object recognition model learning method comprising: a learning information acquisition step of acquiring a captured image, depth information for a captured object included in the captured image, and annotation information indicating the position of a recognition target included in the captured image; and a learning step of learning an object recognition model based on the captured image, depth information, and annotation information.

[0075] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., via wire, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or multiple devices with software.

[0076] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0077] For example, the learning device 100 and the object recognition device 200 according to an embodiment of the present disclosure may function as a computer that performs the learning method and the object recognition method of the present disclosure. Fig. 12 is a diagram showing an example of the hardware configuration of the learning device 100 and the object recognition device 200 according to an embodiment of the present disclosure. The learning device 100 and the object recognition device 200 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0078] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the learning apparatus 100 and the object recognition apparatus 200 may be configured to include one or more of the apparatuses shown in the figures, or may be configured to exclude some of the apparatuses.

[0079] Each function of the learning device 100 and the object recognition device 200 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0080] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the learning unit 103 or the recognition unit 204 described above may be realized by the processor 1001.

[0081] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these programs. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the learning unit 103 or the recognition unit 204 may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be made for other functional blocks. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0082] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be referred to as a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing the learning method and object recognition method according to an embodiment of the present disclosure.

[0083] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0084] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 1004 may include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize at least one of frequency division duplex (FDD) and time division duplex (TDD). For example, the above-mentioned captured image acquisition unit 202 may be realized by the communication device 1004. The communication device 1004 may be implemented with a transmitter and a receiver that are physically or logically separated.

[0085] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0086] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0087] Furthermore, the learning device 100 and the object recognition device 200 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0088] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0089] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0090] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0091] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0092] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0093] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0094] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0095] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0096] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0097] Note that terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Furthermore, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

[0098] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or other corresponding information. For example, a radio resource may be indicated by an index.

[0099] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0100] In this disclosure, the terms "Mobile Station (MS)," "user terminal," "User Equipment (UE)," "terminal," and the like may be used interchangeably.

[0101] A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

[0102] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0103] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0104] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0105] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0106] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0107] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0108] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0109] 100...Learning device, 101...Learning information acquisition unit, 102...Generation unit, 103...Learning unit, 200...Object recognition device, 201...Object recognition model, 202...Captured image acquisition unit, 203...Synthetic captured image acquisition unit, 204...Recognition unit, 250...Image data storage unit, 260...Object recognition model storage unit, 270...Storage unit.

Claims

1. An object recognition model learning device comprising: a learning information acquisition unit that acquires a captured image, depth information for a subject included in the captured image, and annotation information indicating the position of a recognition target included in the captured image; and a learning unit that learns an object recognition model based on the captured image, the depth information, and the annotation information.

2. The object recognition model learning device of claim 1, further comprising a generation unit that generates a composite captured image by combining the captured image with the depth information, and the learning unit that learns an object recognition model based on the captured image, the composite captured image, and the annotation information.

3. The object recognition model learning device according to claim 2, wherein the generation unit generates a synthetic captured image by changing image configuration information that constitutes the captured image in accordance with the specific portion indicated by the depth information.

4. The object recognition model learning device according to claim 2, wherein the learning unit learns based on the captured image, the composite captured image, and the annotation information.

5. The object recognition model learning device according to claim 1, wherein the learning information acquisition unit divides the captured image into an arbitrary grid size and generates a depth image of the arbitrary grid size based on the depth information.

6. An object recognition device comprising: an object recognition model trained based on a captured image, depth information for a subject included in the captured image, and annotation information indicating the position of an object to be recognized included in the captured image; a captured image acquisition unit that acquires the captured image; and a recognition unit that recognizes the object to be recognized using the object recognition model based on the captured image.

7. The object recognition device according to claim 6, further comprising a composite image acquisition unit that acquires a composite image including the depth information for the object contained in the captured image, and the recognition unit inputs the captured image and the composite image into the object recognition model to recognize the object to be recognized.

8. The object recognition device according to claim 7, wherein the recognition unit inputs the captured image into the object recognition model to obtain a recognition result, and recognizes the object to be recognized based on a captured image selected based on the recognition result and a composite captured image obtained from the captured image.

9. The object recognition device according to claim 6, wherein the object recognition model is learned by the object recognition model learning device according to claim 1.

10. An object recognition model learning method comprising: a learning information acquisition step of acquiring a captured image, depth information for a subject included in the captured image, and annotation information indicating the position of a recognition target included in the captured image; and a learning step of learning an object recognition model based on the captured image, the depth information, and the annotation information.

Images