Teacher image data generation device, teacher image data generation method, and program

Abstract

To accurately and efficiently specify an area occupied by an object included in captured image data in an image.SOLUTION: The training data generation device 100 functioning as the object region specifying device includes the processing unit 110, and the processing unit 110 acquires the object information which is information regarding the object from the object included in the captured image data, acquires the shape and the position of the object in the space based on the acquired object information, and specifies the plane figure specified by the object in the image as the region occupied by the object in the image based on the shape and the position of the object.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an object region specifying device, an object region specifying method, a training data generating device, and a program. [Background technology]

[0002] A large amount of training data is required to generate a trained model for machine learning. For example, when performing image recognition, training data must be prepared by performing annotation, which is a process of associating information related to each image data with a large amount of image data containing objects to be recognized. For example, Patent Document 1 describes a manual annotation process for medical images as an example of a process of specifying images of objects to be used as training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-035095 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology in Patent Document 1 has the problem that the selection of the large number of images required to generate training data is largely dependent on manual operation by an operator, which is a huge hassle. Also, even if an attempt is made to automatically select objects contained in images, it is difficult to achieve the required accuracy.

[0005] One object of the present invention is to provide an object area identification device, an object area identification method, a training data generation device, and a program that enable accurate and efficient identification of the area occupied by an object to be recognized within an image. [Means for solving the problem]

[0006] One aspect of the present invention is an object region identification device that includes a processing unit that acquires object information, which is information about an object, from an object included in captured image data, acquires the shape and position of the object in space based on the acquired object information, and identifies a planar figure identified by the object in the image as the region that the object occupies in the image based on the shape and position of the object. [Effects of the Invention]

[0007] According to the present invention, it is possible to accurately and efficiently identify the area within an image occupied by an object to be recognized. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a system configuration diagram illustrating the configuration of a teacher data generation system in one embodiment of the present invention. [Figure 2] 1 is a block diagram illustrating an example of the configuration of hardware and functional blocks of an object region specifying device according to an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram illustrating the configuration of hardware and functional blocks of a marker control unit in one embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of captured image data in the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a camera installation situation in this embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of camera data in the present embodiment. [Figure 7] 1A and 1B are diagrams illustrating an example of modeling of an object in this embodiment. [Figure 8] FIG. 2 is a diagram showing an example of the configuration of object data in the present embodiment. [Figure 9] FIG. 4 is a diagram showing an example of the configuration of object region identification data in this embodiment. [Figure 10]FIG. 2 is a diagram illustrating an example of the configuration of training data according to the present embodiment. [Figure 11] 10 is a flowchart illustrating an example of data processing in a teacher data generation process according to the present embodiment. [Figure 12] 10 is a flowchart illustrating an example of data processing of an object region specifying process in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the present invention will be described based on an embodiment thereof with reference to the accompanying drawings.

[0010] <Teacher Data Generation System 1> Fig. 1 is a system configuration diagram illustrating the system configuration of a teacher data generation system 1 according to one embodiment of the present invention. As shown in Fig. 1, the teacher data generation system 1 according to this embodiment includes a teacher data generation device 100 as an information processing device, and four cameras 200A, 200B, 200C, and 200D. These cameras 200A, 200B, 200C, and 200D are connected to the teacher data generation device 100 via a hub 250. Note that when there is no need to distinguish between the cameras 200A, 200B, 200C, and 200D, they will be simply referred to as cameras 200.

[0011] The teacher data generation device 100 can be connected via a network 400 to other external devices, such as a machine learning system that performs learning using teacher data generated by the teacher data generation system 1, so that they can communicate with each other. The network 400 is a wired or wireless communication line that enables secure communication, such as a dedicated line, a local area network (LAN), the Internet, or a mobile communication network.

[0012] In this embodiment, a use case of the teacher data generation system 1 is assumed to be a case in which a forklift 300 operating in a space S such as a warehouse or freight yard is used as an object to be recognized. That is, the teacher data generation system 1 illustrated in FIG. 1 provides a function for creating teacher data used in a learning process to identify and recognize multiple forklifts 300 captured in an image. Application examples of the present invention are not limited to this, and various situations in which image recognition processing of moving objects in the space S is performed can be envisioned.

[0013] In the teacher data generation system 1 of this embodiment, a marker 310 is attached to the forklift 300 so that the presence of the forklift 300 as an object in an image can be identified. The marker 310 is a light-emitting device equipped with a light-emitting element such as a light-emitting diode (LED). The forklift 300 is provided with a marker control unit 320 to control the light-emitting operation of the marker 310. The operation control of the marker 310 by the marker control unit 320 will be described later with reference to the related drawings. The forklift 300 is also provided with devices for realizing the original functions of the forklift 300, such as handling and transporting goods, but a description of these will be omitted.

[0014] The teacher data generation device 100 acquires and analyzes video data captured by the camera 200 to acquire object information, which is unique information about the forklift 300, based on light emission signals from the markers 310 attached to the forklift 300, which are included in the image data of each frame constituting the video data. The teacher data generation device 100 creates a three-dimensional model of the forklift 300 in space S based on this object information and information about the installation position, imaging direction, and imaging angle of view of each camera 200. Then, based on the three-dimensional model's position in space S, movement direction, and placement relative to the camera imaging direction, the teacher data generation device 100 generates planar graphic information representing the forklift 300 in the image data. The teacher data generation device 100 adds the generated planar graphic information and the corresponding object information about the forklift 300 to each piece of image data and stores the resulting data as teacher data. The teacher data generation device 100 is typically implemented by an electronic device such as a server computer or personal computer equipped with communication capabilities. In the teacher data generation device 100, data processing for generating planar graphic information corresponding to the forklift 300 as an object, in other words, processing for identifying the area occupied by the forklift 300 in an image by image processing such as framing, can be understood as object region identification processing, which is preprocessing of the teacher data generation processing. In this sense, the teacher data generation device 100 in this embodiment includes an object region identification device. The configuration and functions of the teacher data generation device 100 including this object region identification device will be described later with reference to the related drawings.

[0015] The camera 200 is an imaging means having an imaging element such as a CCD and a circuit unit with an image processing function, and has the configuration of a typical digital video camera. Each camera 200 also has a communication circuit unit for communicating with the teacher data generation device 100 via the hub 250. For example, a video camera conforming to the GigE Vision standard can be suitably used as the camera 200, but is not limited to this. A required number of cameras 200 can be installed in the teacher data generation device 100 according to necessary conditions, such as ensuring that there are no blind spots in the space S to be imaged.

[0016] <Teacher data generation device 100> Next, a description will be given of the teacher data generation device 100 according to this embodiment. Fig. 2 is a block diagram illustrating an example of the configuration of hardware and functional blocks of the teacher data generation device 100 according to one embodiment of the present invention.

[0017] 2, the teacher data generation device 100 is an information processing device such as a server computer or a personal computer, and includes a processing unit 110, a memory unit 120, an input / output unit 130, a data interface (data IF) unit 140, a communication unit 150, and an image processing unit 160. The processing unit 110, the memory unit 120, the input / output unit 130, the data IF unit 140, the communication unit 150, and the image processing unit 160 are connected to each other by a bus serving as an internal communication line (not shown).

[0018] The processing unit 110 is an arithmetic unit configured with a processor such as a CPU, and reads and executes various programs and data from the storage unit 120 (described later) to realize the functions of the teacher data generation device 100. In this embodiment, the processing unit 110 executes data processing for each of the functional units: the image registration unit 111, the target image extraction unit 112, the marker signal analysis unit 113, the modeling unit 114, the movement direction identification unit 115, the target range identification unit 116, the additional information attachment unit 117, and the teacher data registration unit 118. The operation of each functional unit will be described later.

[0019] The storage unit 120 is a storage area for various programs and various data for causing the hardware group to function as the teacher data generation device 100, and can be configured with a ROM, RAM, flash memory, a solid-state drive (SSD), a hard disk (HDD), etc. Specifically, the storage unit 120 stores a program for causing the processing unit 110 to execute each function of this embodiment (a control program for the teacher data generation device 100), various parameters, operation input data input from outside, captured image data captured by the camera 200, target range data for identifying an object area in an image, generated teacher data, etc.

[0020] The input / output unit 130 is composed of various input devices such as a keyboard, mouse, touch panel, microphone, etc. that enable data input to the teacher data generation device 100 from outside, and output devices such as a monitor display and speaker for outputting and displaying image data received from the camera 200, target range data, image data including teacher data, etc.

[0021] The data IF unit 140 has a function of controlling data communication between the processing unit 110 , the storage unit 120 , the input / output unit 130 , the communication unit 150 , and the image processing unit 160 .

[0022] The communication unit 150 is a communication module that transmits and receives various data such as image data and marker signal data to and from the camera 200, and is configured as hardware such as a network interface card (NIC).

[0023] Image processing unit 160 has the function of performing distortion correction, color adjustment, and noise removal on the digital data of frames output from imaging unit 210 of camera 200, and outputting the result to processing unit 110. Digital data of frames generated in time series by imaging unit 210 of camera 200 is input to image processing unit 160. Imaging unit 210 of camera 200 generates a plurality of captured images captured at different timings as a moving image using an imaging device such as a charge-coupled device (CCD), and converts the image signals into digital data to generate frames.

[0024] Next, the functions of the teacher data generation device 100 realized by the programs executed by the processing unit 110 will be described.

[0025] The image registration unit 111 has a function of performing a process of assigning an image ID, which is an identification code for making each frame data included in the moving image data input from the camera 200 identifiable, and storing the data as captured image data in the captured image data storage unit 121 of the storage unit 120. Fig. 4 shows an example of captured image data stored in the captured image data storage unit 121. Each piece of captured image data shown in Fig. 4 is associated with an image ID, and the image data, camera data which is information indicating the status of the camera 200 that captured the image data, and the capture time are recorded. The captured image data storage unit 121 stores a plurality of pieces of captured image data shown in Fig. 4 in chronological order.

[0026] Here, camera data will be described with reference to FIGS. 5 and 6. FIG. 5 is a schematic diagram illustrating the installation status of cameras in this embodiment, and FIG. 6 is a diagram illustrating an example of camera data. In this embodiment, each camera 200 is installed so as to be suspended from a ceiling surface (not shown) and is arranged so as to overlook the space S from above. In the example of FIG. 5, the depression angle of the camera 200 is represented by θ, and the planar angle with respect to an arbitrary reference coordinate axis set in the space S is represented by φ. The imaging angle of view of the camera 200 is α. The installation position of the camera 200 is represented by three-dimensional coordinates (x, y, z) in the space S. In this case, the camera data in this embodiment includes the camera position (x, y, z), the combination of the angle and depression angle (φ, θ) from the reference axis indicating the camera's imaging direction, and the imaging angle of view α, recorded in association with a camera ID=n, which is an identification code for uniquely identifying each camera 200. This camera data is used in processing to identify the area occupied in the image by a forklift 300, an object included in the image. This camera data can be attached to the video data input from each camera 200, for example.

[0027] The target image extraction unit 112 extracts captured image data to be used for generating teacher data based on whether the captured image data stored in chronological order in the captured image data storage unit 121 by the image registration unit 111 contains an image of the marker 310.

[0028] The marker signal analysis unit 113 analyzes the time-series images of the marker 310 contained in the captured image data stored in chronological order in the captured image data storage unit 121, thereby obtaining object information regarding the forklift 300 on which the marker 310 is installed.

[0029] Here, we will explain the marker signal, which is an optical emission signal emitted by the marker 310 installed on the forklift 300. Fig. 3 is a block diagram illustrating the hardware and functional block configuration of the marker control unit 320. As described with reference to Fig. 1, in this embodiment, the marker control unit 320 is installed on the forklift 300.

[0030] 3, the marker control unit 320 is configured as an information processing device equipped with a processor such as a CPU, and includes a processing unit 321, a storage unit 322, a communication unit 323, and a driver 324. The processing unit 321, the storage unit 322, the communication unit 323, and the driver 324 are connected to each other by a bus serving as an internal communication line (not shown).

[0031] The processing unit 321 functions as a light emission control unit 3211 by executing a corresponding program stored in the memory using a processor.

[0032] The light emission control unit 3211 controls the blinking of the light emitting element of the marker 310 via the driver 324 to make the marker 310, which is a light emitting device, function as a signal transmission device. The marker 310 generates a light emission pattern representing the object information by controlling the blinking of the light emitting element, in order to transmit object information related to the forklift 300 on which the marker 310 is installed to the teacher data generation device 100 via the camera 200.

[0033] That is, the light emission control unit 3211 reads out the object information of the forklift 300 stored in the object information storage unit 3221 of the storage unit 322.

[0034] The light emission control unit 3211 determines a light emission pattern that changes the emitted color in a time series in accordance with the object information that is communication data.

[0035] The light emission control unit 3211 outputs information about the light emission pattern to the driver 324. The driver 324 generates a drive signal for changing the hue of the light emitted by the marker 310 over time, according to the information about the light emission pattern from the light emission control unit 3211. The marker 310 emits light whose hue changes over time, according to the drive signal output from the driver 324. For example, the emitted light color is one of the three primary colors, and can be any of red (R), green (G), or blue (B), which are colors in the wavelength bands used for color modulation in visible light communication.

[0036] Returning to the explanation of Figure 2, the marker signal analysis unit 113 uses the image of the marker 310 contained in the captured image data stored in chronological order in the captured image data storage unit 121 to read the light emission pattern of the marker 310 and obtain the object information represented by that light emission pattern.

[0037] The object information will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram that schematically expresses object information about a forklift 300, which is an object in this embodiment, and Fig. 8 is a diagram that shows an example of the object information.

[0038] In this embodiment, the forklift 300 as an object is represented as a rectangular parallelepiped model as shown in FIG. 7. This rectangular parallelepiped is set so as to circumscribe the outer edge of the target forklift 300. In the example of FIG. 7, the dimensions of this rectangular parallelepiped model are width W, depth D, and height H. The object information also includes the installation position of the marker 310 relative to the forklift 300. In the example of FIG. 7, three-dimensional local coordinates are set with the vertex on the front right side of the illustrated rectangular parallelepiped model as the origin O. In the example of FIG. 7, the installation position of the marker 310 on this coordinate system is expressed as (a, b, 0). The object information shown in the example of FIG. 8 includes, using a marker ID, which is an identification code for uniquely identifying the marker 310, as a key, the object's width W (X-direction dimension), object's depth (Y-direction dimension), object's height H (Z-direction dimension), marker position (a, b, 0), object type information, and the object's movement direction. In the example of Fig. 8, the type of object is recorded as a forklift and the movement direction is recorded as forward. Here, the movement direction is information for distinguishing whether the forklift 300 as an object is moving forward or backward, and can be configured so that the marker control unit 320 provided in the forklift 300 acquires this information from the drive control system of the forklift 300, for example.

[0039] 8 as object information may be assigned an object ID, which is an identification code for identifying each object, in advance for each object, and stored in association with the object ID in the database of storage unit 120. In this case, when placing marker 310 on forklift 300, which is an object, the marker ID of marker 310 may be associated with the object ID of forklift 300 on which marker 310 is placed. By doing so, processing unit 110 can search the database of storage unit 120 using the marker ID received from forklift 300 as a key to obtain information such as the dimensions and type of the forklift on which marker 310 having the received marker ID is placed.

[0040] Returning to the explanation of Figure 2, the modeling unit 114, based on the object information received from the marker 310, determines the shape and dimensions of a rectangular parallelepiped modeling the forklift 300, which is an object present in the space S, and the position of the rectangular parallelepiped based on the marker position. Note that the three-dimensional shape representing the object is not limited to a rectangular parallelepiped, and may be, for example, a cone, a spheroid, or the like, as appropriate, depending on the specific shape and dimensions of the object to be recognized.

[0041] The movement direction identification unit 115 has a function of identifying the movement direction of the forklift 300 on which the marker 310 is installed based on changes in the position of the marker 310 in the image contained in the time-series captured image data. This movement direction can be determined by acquiring the position of the marker 310 in the captured image data stored in time series as coordinates in the image and converting a vector representing the change in the coordinates into a vector in the space S based on the imaging direction and imaging angle of view of the camera 200. In this embodiment, in a rectangular parallelepiped model representing the forklift 300 as an object, a plane perpendicular to the movement direction of the forklift 300 is used to determine a region for identifying the object (forklift 300) in the image. Note that the movement direction determined by the movement direction identification unit 115 based on the captured image data is the movement direction in the space S based on changes in the position of the object in the image. Therefore, this is a separate concept from the movement direction of the object itself described with respect to the example of object information in FIG. 8 , or in this embodiment, the "type of forward or backward movement of the forklift 300."

[0042] The target range identification unit 116 has a function of executing data processing to identify the area in the image occupied by the forklift 300, which is an object, based on the rectangular parallelepiped model representing the forklift 300, which is obtained by the modeling unit 114, the arrangement state of the rectangular parallelepiped model in the space S, and the movement direction of the forklift 300, which is an object, in the space S, which is obtained by the movement direction identification unit 115. Specifically, the target range identification unit 116 rotates the rectangular parallelepiped model, which is obtained in the space S, so that it aligns with the known movement direction. The rotation direction and rotation angle at this time can be determined, for example, so that the depth direction (Y direction) of the rectangular parallelepiped model coincides with the movement direction.

[0043] In order to represent the object surface, which has been identified as a surface orthogonal to the movement direction of the rectangular parallelepiped model, as a planar graphic for identifying the area occupied by the object in the image, the target range identification unit 116 performs projective transformation of the rectangular parallelepiped model aligned with the movement direction onto a plane orthogonal to the imaging direction based on the imaging direction of the camera 200. By performing projective transformation onto this plane, a rectangle, which is a planar graphic representing the area occupied by the object in the image, can be obtained.

[0044] Target range identification unit 116 calculates the position coordinates of the rectangle within the image based on the imaging angle of view of camera 200, generates data by adding rectangle information including the calculated position of the rectangle, the imaging angle of view of camera 200, and object information identified by marker 310 to the image data, and stores this as object region identification data in object region identification data storage unit 122, thereby completing the process of identifying the object region within the image. Fig. 9 shows an example of the configuration of object region identification data.

[0045] The additional information adding unit 117 has a function of adding additional information to generate training data based on the object region identification data. Specifically, the additional information adding unit 117 has a function of adding necessary additional information when a portion of the image region of the forklift 300, which is an object in an image, is visually blocked by another object (occlusion occurs) or when a portion of the image region of the forklift 300, which is an object, is missing at the edge of the image (a so-called "cut-off state"). Specifically, when the additional information adding unit 117 determines that an occlusion or an object missing at the edge of the screen has occurred, it performs a process to complement the occlusion of the object or the missing portion at the edge of the screen based on captured image data captured by another camera 200 at the same time as the captured image data being processed. Known image processing techniques used in 3D image processing can be appropriately adopted for the image processing to complement the occlusion of the object or the missing portion at the edge of the screen.

[0046] The teacher data registration unit 118 stores the object region identification data, on which the additional information addition unit 117 has performed a compensation process for occlusion or partial loss at the screen edge of the object as necessary, as teacher data in the teacher data storage unit 123. Fig. 10 shows an example of the configuration of the teacher data stored in the teacher data storage unit 123.

[0047] The above-described functions of the processing unit 110 of the teacher data generation device 100 make it possible to accurately identify the area within an image occupied by an object included in captured image data. In the processing unit 110 of the teacher data generation device 100, the parts that perform the data processing to achieve this, namely the target image extraction unit 112, the marker signal analysis unit 113, the modeling unit 114, the movement direction identification unit 115, and the target range identification unit 116, can be understood as an "object region identification device" that provides the prerequisite functions for generating appropriate teacher data. By performing the above-described additional information addition process as necessary on the object region identification data generated by the object region identification device, it is possible to efficiently generate appropriate teacher data.

[0048] <Teacher data generation process and object region identification process> The teacher data generation process performed by the teacher data generation device 100 having the above configuration and functions, and the object area identification process performed in the teacher data generation device 100 as a prerequisite for the teacher data generation process will be described with reference to a flowchart showing an example of data processing.

[0049] <<Teacher data generation process>> 11 is a flowchart showing an example of data processing executed by the teacher data generation device 100 in this embodiment. The teacher data generation device 100 starts the data processing exemplified in FIG. 11 when an instruction to start processing is input via the input / output unit 130.

[0050] In step S1, the processing unit 110 of the teacher data generation device 100 receives an object ID, which is information for identifying an object to be subjected to the teacher data generation process, in this embodiment, a forklift 300 moving within the space S. Specifically, the object ID is information for identifying a marker 310 attached to the forklift 300 as an object.

[0051] In step S2, the processing unit 110 acquires the captured image data stored in the captured image data storage unit 121 in chronological order.

[0052] In step S3, the processing unit 110 executes processing for detecting the marker 310 in the acquired captured image data. Specifically, the processing unit 110 detects, for example, an image area in the image that has a certain level of brightness or higher as the marker 310.

[0053] In step S4, the processing unit 110 executes an object region specifying process on the captured image data. The object region specifying process in step S4 will be described later with reference to FIG.

[0054] In step S5, the additional information adding unit 117 determines whether there is an object blocking part of the object in the image, in other words, whether part of the object is hidden. If it is determined that part of the object is hidden (YES in step S5), in step S6, the additional information adding unit 117 executes a process to complement the hidden part of the object based on captured image data captured at the same time by another camera 200. If it is determined that the object is not hidden (NO in step S5), the additional information adding unit 117 proceeds to the process of step S7.

[0055] In step S7, additional information adding unit 117 determines whether any part of the object is missing at the edge of the screen. If it is determined that any part of the object is missing at the edge of the screen (YES in step S7), in step S8, additional information adding unit 117 executes a process of adding information indicating that the missing part of the object is at the edge of the screen, based on captured image data captured at the same time by another camera 200. If it is determined that the object is not missing at the edge of the screen (NO in step S7), the process proceeds to step S9.

[0056] In step S9, the processing unit 110 stores the training data based on the object region specifying data generated in step S4 in the training data storage unit 123, and then ends the process.

[0057] According to the above-described training data generation process, a large amount of training data to be used for learning to perform image recognition of the forklift 300 as an object can be generated accurately and efficiently.

[0058] <<Object area identification processing>> Fig. 12 is a flowchart showing an example of data processing of the object region identification process executed in the teacher data generation device 100 in this embodiment. This data processing corresponds to the processing of step S4 in the teacher data generation process exemplified in Fig. 11. In the object region identification process, the following processing is repeatedly executed for the captured image data in which the marker 310 was detected in step S3 in Fig. 11 (loop processing of step S11).

[0059] In step S12, the target image extraction unit 112 determines whether the captured image data contains an object ID designated as a processing target. If it is determined that there is no object ID to be processed (NO in step S12), the target image extraction unit 112 proceeds to processing the next captured image data. If it is determined that there is an object ID to be processed (YES in step S12), in step S13, the marker signal analysis unit 113 receives corresponding object information based on the marker signal. As described above, this object information includes the shape and dimensions of the forklift 300, which is the object, and the installation position of the marker 310.

[0060] In step S14, the marker signal analysis unit 113 acquires the position of the marker 310 in the space S as the position of the object, based on the position of the marker 310 in the captured image and the imaging direction and imaging angle of view of the camera 200.

[0061] In step S15, the movement direction identification unit 115 calculates the change in the object's position in space S acquired last time and the object's position acquired this time based on the change in the position of the marker 310 placed on the object, and identifies the movement direction of the object within space S.

[0062] In step S16, the modeling unit 114 generates a rectangular parallelepiped model that circumscribes the forklift 300, which is the object, based on the object information received in step S13.

[0063] In step S17, the target range specification unit 116 rotates the generated rectangular parallelepiped model so that it is aligned with the specified movement direction within the space S. Specifically, the target range specification unit 116 rotates the rectangular parallelepiped model so that its depth direction (Y direction) coincides with the movement direction.

[0064] In step S18, the target range identification unit 116 performs projective transformation of the plane perpendicular to the movement direction of the rectangular parallelepiped model onto a plane perpendicular to the imaging direction of the camera 200. As a result, a rectangle, which is a plane perpendicular to the movement direction of the rectangular parallelepiped model representing the object in the image, is identified as planar graphic information representing the area occupied by the object in the image.

[0065] In step S19, the target range identification unit 116 calculates the position of the rectangle in the image based on the imaging angle of view of the camera 200. This process makes it possible to draw rectangular information (planar graphic information) that identifies the object region in the image.

[0066] In step S20, the target range identification unit 116 associates the rectangle information with the imaging angle of view of the camera 200 and the corresponding object information, stores the result as object region identification information in the object region data storage unit 122, and repeats the above process until the loop process of step S11 ends. When the loop process of step S11 ends, the process returns to the main routine, the teacher data generation process of FIG. 11.

[0067] According to the object area identification process of the present embodiment described above, the area occupied by the forklift 300 as an object contained in the captured image data can be identified accurately and efficiently, and this can be used to efficiently generate a large amount of training data for training the image recognition process of the object.

[0068] The shape of the forklift 300 as an object may be set as a polyhedron that circumscribes the forklift 300 .

[0069] In this way, the external dimensions of the forklift 300 to be recognized can be appropriately modeled.

[0070] The processing unit 110 may determine the movement direction of the object based on a change in the position of a marker 310 attached to the forklift 300 as an object in the acquired multiple captured image data, and control the placement state of the object in space based on the determined movement direction.

[0071] In this way, for the forklift 300 as an object, a surface for identifying the object region can be identified according to the direction of movement of the forklift 300.

[0072] The processing unit 110 may rotate the object in the captured image so that it faces the imaging direction of the object based on the positional state of the object identified based on the movement direction of the object and the imaging direction of the captured image data, and obtain a planar figure for identifying the area of ​​the object in the captured image by projecting a two-dimensional figure that faces the imaging direction of the object onto a plane in the space that is perpendicular to the movement direction.

[0073] In this way, a plane figure that accurately represents the area occupied by objects in various positions and arrangements within the image can be acquired, and the object in the image can be framed.

[0074] The object information may include an object ID, which is information for identifying the object.

[0075] In this way, the processing unit 110 can easily identify and recognize the forklift 300 as an object.

[0076] The object information may include information about the shape of the object and the installation position of the transmitting means on the object.

[0077] In this way, the position of the forklift 300, which is the object, can be easily acquired as the position of the transmitting means.

[0078] The transmitting means may be a light emitting device, and the object information may be transmitted by a light emitting signal.

[0079] In this way, object information can be transmitted through the camera 200 that captures the image without providing a separate communication means.

[0080] The shape of the object and the installation position of the transmitting means on the object may be set in advance in accordance with the object information.

[0081] In this way, it is possible to create a three-dimensional model of the object and place it in the space based on the received object information.

[0082] The transmitting means may be an LED marker device that transmits a light emitting signal including the object information.

[0083] In this way, object information can be transmitted using a simple method.

[0084] This embodiment can be configured as a teacher data generation device that includes the object region identification device, adds planar graphic information indicating the object and attribute information related to the object to the captured image data, and generates teacher data to be used in learning processes related to recognition of the object.

[0085] In this way, a large amount of training data for learning image recognition processing of an object such as the forklift 300 can be efficiently generated.

[0086] The above-described series of processes can be executed by hardware or software. In other words, the functional configurations shown in Figures 2 and 3 are merely illustrative and are not particularly limited. That is, it is sufficient for the training data generation device 100, which also serves as an object region identification device, to have the functionality to execute the above-described series of processes as a whole. The functional blocks used to realize these functions are not particularly limited to the examples shown in Figures 2 and 3. Furthermore, a single functional block may be configured as a hardware unit, a software unit, or a combination thereof. The functional configuration in this embodiment is realized by a processor that executes arithmetic processing. Processors that can be used in this embodiment include those configured as various processing units, such as single processors, multiprocessors, and multicore processors, as well as those that combine these various processing units with processing circuits, such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays).

[0087] When a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer that can execute various functions by installing various programs, such as a general-purpose personal computer.

[0088] The recording medium containing such a program may be a removable medium such as a USB memory that is distributed separately from the device main body in order to provide the program to the user, or may be a recording medium that is provided to the user in a state where it is pre-installed in the device main body. Removable media may be, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. Optical disks may be, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, etc. Magneto-optical disks may be, for example, MDs (Mini-Disks), etc. Furthermore, recording media that are provided to the user in a state where they are pre-installed in the device main body may be, for example, a ROM on which the program is recorded, or a hard disk included in the storage unit 120, 322, etc.

[0089] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually.

[0090] Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and it is also possible to combine the configurations of the above embodiments and modifications. Furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included within the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0091] The inventions described in the claims of the present application as originally filed are set forth below. [Appendix 1] Obtaining object information, which is information about an object, from the object included in the captured image data; acquiring a shape and a position of the object in space based on the acquired object information; a processing unit for specifying a plane figure specified by the object in the image as an area occupied by the object in the image based on the shape and position of the object; Object region identification device. [Appendix 2] 2. The object region identifying device according to claim 1, wherein the shape of the object is set as a polyhedron circumscribing the object. [Appendix 3] 3. The object region identification device according to claim 1, wherein the processing unit identifies a movement direction of the object based on a change in position of a transmitting means installed on the object in the acquired plurality of pieces of captured image data, and controls a placement state of the object within the space based on the identified movement direction. [Appendix 4] 4. The object region identification device according to claim 1, wherein the processing unit rotates the object in the captured image so as to face the imaging direction of the object based on a positional state of the object identified based on the movement direction of the object and the imaging direction of the captured image data, and performs projective transformation on the two-dimensional figure that faces the imaging direction of the object onto a plane in the space that is orthogonal to the movement direction, thereby acquiring a planar figure for identifying the region of the object in the captured image. [Appendix 5] 5. The object region specifying device according to any one of appendices 1 to 4, wherein the object information includes object identification information that is information for identifying the object. [Appendix 6] 6. The object region specifying device according to any one of appendices 1 to 5, wherein the object information includes information about the shape of the object and the installation position of the transmitting means on the object. [Appendix 7] 7. The object region specifying device according to any one of claims 3 to 6, wherein the transmitting means is a light emitting device and transmits the object information by a light emitting signal. [Appendix 8] 8. The object region identification device according to any one of appendices 3 to 7, wherein the shape of the object and the installation position of the transmitting means on the object are set in advance in accordance with the object information. [Appendix 9] 9. The object region identification device according to any one of appendices 3 to 8, wherein the transmitting means is an LED marker device that transmits an illuminating signal including the object information. [Appendix 10] An object region specifying device according to any one of Supplementary Notes 1 to 9, A training data generation device that adds planar graphic information representing the object and attribute information relating to the object to the captured image data, and generates training data to be used in a learning process for recognizing the object. [Appendix 11] The information processing device Obtaining object information, which is information about an object, from the object included in the captured image data; acquiring a shape and a position of the object in space based on the acquired object information; Identifying a plane figure identified by the object in the image based on the shape and position of the object as an area occupied by the object in the image; Object region identification method. [Appendix 12] In the information processing device, A process of acquiring object information, which is information about an object, from an object included in the captured image data; A process of acquiring a shape and a position of the object in space based on the acquired object information; a process of identifying a plane figure identified by the object in the image based on the shape and position of the object as an area occupied by the object in the image; A program that executes the following. [Explanation of symbols]

[0092] 1. Teacher data generation system 100 Teacher data generation device 110 Processing section 111 Image Registration Unit 112 Target image extraction unit 113 Marker signal analysis unit 114 Modeling Department 115 Movement direction identification part 116 Target Scope Identification Section 117 Additional information assignment section 118 Teacher Data Registration Department 120 Storage section 130 Input / output section 140 Data Interface Unit 150 Communications Department 160 Image processing section 121 Captured image data storage unit 122 Object area identification data storage unit 123 Teacher data storage unit 200A, 200B, 200C, 200D Camera 210 Imaging unit 300 Forklift (Object) 310 Marker 320 Marker control unit 321 Processing Section 3211 Light Emission Control Unit 322 Storage section 3221 Object Information Storage Unit 323 Communications Department 324 Driver 400 Network S space

Claims

1. Obtaining object information, which is information about an object, from the object included in the captured image data; acquiring a shape and a position of the object in space based on the acquired object information; a processing unit for specifying a plane figure specified by the object in the image as an area occupied by the object in the image based on the shape and position of the object; Object region identification device.

2. The object region specifying device according to claim 1 , wherein the shape of the object is set as a polyhedron circumscribing the object.

3. 3. The object region identification device according to claim 1, wherein the processing unit identifies a movement direction of the object based on a change in a position of a transmitting means installed on the object in the acquired plurality of pieces of captured image data, and controls a placement state of the object within the space based on the identified movement direction.

4. 4. The object region identification device according to claim 3, wherein the processing unit rotates the object in the captured image so as to face the imaging direction of the object based on a positional state of the object identified based on the movement direction of the object and on the imaging direction of the captured image data, and performs projective transformation on a two-dimensional figure facing the imaging direction of the object onto a plane in the space that is orthogonal to the movement direction, thereby acquiring a planar figure for identifying the region of the object in the captured image.

5. The object region specifying device according to claim 1 , wherein the object information includes object identification information that is information for identifying the object.

6. The object region specifying device according to claim 3 , wherein the object information includes information about a shape of the object and an installation position of the transmitting means on the object.

7. 7. The object region specifying device according to claim 3, wherein the transmitting means is a light emitting device, and transmits the object information by a light emitting signal.

8. The object region specifying device according to claim 3 , wherein a shape of the object and an installation position of the transmitting means on the object are set in advance in accordance with the object information.

9. 9. The object region specifying device according to claim 3, wherein the transmitting means is an LED marker device that transmits a light emitting signal including the object information.

10. The object region specifying device according to any one of claims 1 to 9, A training data generation device that adds planar graphic information representing the object and attribute information relating to the object to the captured image data, and generates training data to be used in a learning process for recognizing the object.

11. The information processing device Obtaining object information, which is information about an object, from the object included in the captured image data; acquiring a shape and a position of the object in space based on the acquired object information; Identifying a plane figure identified by the object in the image based on the shape and position of the object as the area that the object occupies in the image; Object region identification method.

12. In the information processing device, A process of acquiring object information, which is information about an object, from an object included in the captured image data; A process of acquiring a shape and a position of the object in space based on the acquired object information; a process of identifying a plane figure identified by the object in the image based on the shape and position of the object as an area occupied by the object in the image; A program that executes the following.

Images