INFORMATION PROCESSING EQUIPMENT, PRODUCTION PROCESS AND PRODUCTION PROGRAM

Abstract

Information processing facility (100), comprising: a capture unit (120) that captures a foreground image to which a foreground image angle is assigned and which displays a target object (20), a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position at the time the background image is generated; an overlay position determination unit (141) that determines an overlay position in the background image; a computation unit (142) that computes a plane as an area containing the overlay position using the point cloud data and computes a background image angle as an angle between a straight line connecting the camera position and the overlay position and the computed plane; a transformation unit (145) that performs a projective transformation on the foreground image in the image acquisition direction in the background image such that a camera position corresponding to the foreground image in a world coordinate system becomes equivalent to a camera position corresponding to the background image in the world coordinate system; and a superimposition unit (146) which overlays the foreground image onto the background image to create a combined image, wherein the foreground image angle is an angle between a plane on which the target object (20) was positioned when an original foreground image was produced and a straight line representing an image acquisition direction as a direction in which the image of the target object (20) is acquired, and is an angle which is equal to or close to the background image angle.

Description

AREA OF TECHNOLOGY

[0001] The present disclosure relates to an information processing device, a generation method and a generation program. TECHNICAL BACKGROUND

[0002] It is known that object recognition is performed using a learned model. During the learning phase of the learned model, a large number of images are required. It is possible to prepare a large number of images containing a general target object, such as a person, a car, or an animal. For example, it is possible to generate a large number of images of a general target object using an open-source tool. However, it is difficult to generate a large number of images of a specific target object, such as a particular vehicle, plant, or product. Furthermore, if the attitude of the target object changes, it is difficult to prepare a large number of images of target objects in different attitudes. Under these circumstances, a technology for generating such images has been proposed (see patent reference 1).An information processing device according to patent specification 1 captures three-dimensional information containing a multitude of images, selects a recognition target from the three-dimensional information, generates a multitude of foreground images from the multitude of images by extracting an image of a recognition target area as an area corresponding to the selected recognition target from each of the multitude of images, and generates a multitude of combined images by combining each of the multitude of foreground images with each of a multitude of background images. STATE OF THE TECHNOLOGY PATENT REFERENCE

[0003] Patent reference 1: WO 2020 / 152763 SUMMARY OF THE INVENTION; PROBLEM TO BE SOLVED BY THE INVENTION

[0004] When overlaying a foreground image with a background image, there are cases where unrealistic data is generated because no restrictions have been applied. For example, an image might be created in which the target object is positioned at an unrealistic angle or similar. If such images are used as training data, the recognition accuracy of the learned model decreases.

[0005] One objective of this disclosure is to generate realistic data. MEANS TO SOLVE THE PROBLEM

[0006] An information processing facility in accordance with one aspect of the present disclosure is provided.The information processing unit includes a capture unit that captures a foreground image, to which a foreground image angle is assigned and which displays a target object, a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position at the time the background image was generated; an overlay position determination unit that determines an overlay position in the background image; a calculation unit that calculates a plane as an area including the overlay position using the point cloud data and calculates a background image angle as an angle between a straight line connecting the camera position and the overlay position and the calculated plane; and an overlay unit that overlays the foreground image onto the background image.The foreground image angle is an angle between a plane on which the target object lay when an original foreground image was created, and a straight line representing an image capture direction as the direction in which the image of the target object is captured, and is an angle that is equal to or close to the background image angle. IMPACT OF THE INVENTION

[0007] According to the present disclosure, realistic data can be generated. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a schematic representation of the hardware of an information processing facility. Fig. Figure 2 is a block diagram showing the functions of the information processing unit. Fig. Figure 3 is a representation showing an example (No. 1) of a method for capturing images of the target object. Fig. Figure 4 is a representation showing an example (No. 2) of the procedure for capturing images of the target object. Fig. Figure 5 is a representation showing an example (No. 3) of the procedure for capturing images of the target object. Fig. Figure 6 shows an example of the case where an object moves in a constant direction. Fig. Figure 7 is a representation showing an example (No. 4) of the procedure for capturing images of the target object. Fig. Figure 8 is a representation showing an example of an angle θ. Fig. Figure 9 is a representation that shows an example of a measurement table. Fig. Figure 10 is a block diagram showing the functions of a foreground image generation unit. Fig. Figure 11 is a representation showing an example (No. 1) of an image. Fig. 12(A) and Fig. 12(B) are illustrations showing an example (No. 2) of the image. Fig. 13(A) and Fig. 13(B) are illustrations showing an example (No. 3) of the image. Fig. Figure 14 is a flowchart showing an example of a process performed by the foreground image generation unit. Fig. Figure 15 is a representation showing an example of a camera coordinate system. Fig. Figure 16 is a representation showing the functions of a data generation unit. Fig. Figure 17 is a flowchart showing an example (No. 1) of a process performed by an overlay position determination unit and a computation unit. Fig. Figure 18 is a flowchart showing an example (No. 2) of the process performed by the overlay position determination unit and the computation unit. Fig. Figure 19 is a flowchart showing an example (No. 1) of a superposition process. Fig. Figure 20 is a flowchart showing example (No. 2) of the superposition process. Fig. 21(A) and Fig. 21(B) are illustrations that show an example of a case in which the appearance varies. Fig. Figure 22 is a representation showing an example of a coordinate system for setting the object position. Fig. 23(A) and Fig. 23(B) are illustrations (No. 1) to explain the projective transformation. Fig. Figure 24 is a representation (No. 2) to explain the projective transformation. Fig. Figure 25 is a representation showing an example of an image after a scaling transformation. Fig. Figure 26 is a representation showing the transition from projective transformation to superposition. MODE FOR EXECUTING THE INVENTION

[0008] One embodiment is described below with reference to the drawings. The following embodiment is only an example, and a variety of modifications are possible within the scope of this disclosure. embodiment

[0009] Fig. Figure 1 is a schematic representation of the hardware of an information processing device. The information processing device 100 is a device that executes a generation process. The information processing device 100 is, for example, a server, a personal computer (PC), a smartphone, or similar.

[0010] The information processing device 100 comprises a processor 101, a volatile memory device 102, a non-volatile memory device 103, an input device 104 and a display device 105. It is also permissible if the input device 104 and the display device 105 are located outside the information processing device 100.

[0011] Processor 101 controls the information processing unit 100 as a whole. Processor 101 could be, for example, a central processing unit (CPU), a field-programmable gate array (FPGA), a graphics processing unit (GPU), or similar. Processor 101 can also be a multiprocessor. The information processing unit 100 may also include a processing circuit.

[0012] The volatile storage device 102 is a main memory of the information processing device 100. The volatile storage device 102 is, for example, random access memory (RAM). The non-volatile storage device 103 is an auxiliary memory of the information processing device 100. The non-volatile storage device 103 is, for example, a hard disk drive (HDD) or a solid-state drive (SSD).

[0013] The input device 104 is a keyboard, a touch panel, or similar. The display device 105 is a display.

[0014] The following describes the functions of the Information Processing Unit 100.

[0015] Fig. Figure 2 is a block diagram showing the functions of the information processing unit. The information processing unit 100 comprises a storage unit 110, a capture unit 120, a foreground image generation unit 130, and a data generation unit 140.

[0016] The memory unit 110 can be implemented as a memory area reserved in the volatile memory device 102 or in the non-volatile memory device 103.

[0017] The acquisition unit 120, the foreground image generation unit 130, and the data generation unit 140 can be implemented partially or entirely by processing circuits. Furthermore, the acquisition unit 120, the foreground image generation unit 130, and the data generation unit 140 can be implemented wholly or partially as modules of a program executed by the processor 101. The program executed by the processor 101 is also referred to, for example, as the generation program. The generation program may, for example, be recorded on a recording medium.

[0018] The storage unit 110 stores a variety of information. For example, the storage unit 110 stores a measurement table. The measurement table will be described later.

[0019] The acquisition unit 120 captures images of a target object 20. The acquisition unit 120 captures the images, for example, from a camera. The camera is, for example, a red-green-blue depth camera (RGB-D). If the camera is an RGB camera, the acquisition unit 120 can capture the images from a combination of the RGB camera and a sensor (e.g., an IMU sensor (Inertial Measurement Unit), an infrared sensor, or a LiDAR (Light Detection and Ranging)) that is capable of measuring the distance between the RGB camera and the target object 20.

[0020] It is also possible that the capture unit 120 captures the images of target object 20 from an external device. The external device could be, for example, a cloud server. The depiction of the external device is omitted.

[0021] The following is an example of a method for capturing images of target objects.

[0022] Fig. Figure 3 is a representation showing an example (No. 1) of the procedure for capturing images of the target object. Fig. Figure 3 shows a robot arm 10. A camera 11 is attached to one end of the robot arm 10.

[0023] For example, the installation position of the robot arm 10 is used as a starting point. The images generated by the camera 11 are processed around the origin using a world coordinate system.

[0024] Fig. Figure 4 is a representation showing an example (No. 2) of the procedure for capturing images of the target object. Fig. Figure 4 shows the target object 20. The target object 20 is arranged on a plane on which the robot arm 10 is installed. Fig. Figure 4 shows a target object setting point 21. The target object setting point 21 is a point that is in contact with the plane and is a barycenter or center of the target object 20. Therefore, the Z-coordinate of the target object setting point 21 is 0. Furthermore, the target object setting point 21 can also be determined before the image is taken. The target object setting point 21 can also be set so that a central point of the camera 11 and a central point of the target object 20 coincide.

[0025] For example, an internal parameter of camera 11 can be determined through camera calibration.

[0026] The following section describes two methods for capturing images in detail. First, a first method for capturing images is shown.

[0027] Fig. Figure 5 is a diagram showing an example (No. 3) of the method for capturing images of the target object. The robot arm 10 captures images of the target object 20 from different angles while moving. Furthermore, the time of image capture, the image capture area, and similar parameters can be changed accordingly.

[0028] Next, a second method for image acquisition is described. In this second method, image acquisition is carried out using the following technique.

[0029] Fig. Figure 6 shows an example of the case where an object moves in a constant direction. Fig. Figure 6 shows a conveyor belt 30. An object 31 has been placed on the conveyor belt 30. The object 31 moves in a constant direction. A camera 32 records images of the linearly moving object 31. The images of the target object 20 are captured using this image acquisition method. The method is described below using the following examples: Fig. 7 specifically described.

[0030] Fig. Figure 7 is a diagram showing an example (No. 4) of the procedure for capturing images of the target object. The robot arm 10 captures images of the target object 20 while moving linearly.

[0031] Through the image acquisition of target object 20 described above, the information processing unit 100 is able to acquire images of target object 20. The method for acquiring the images of target object 20 can, incidentally, also be a different method than the one described above.

[0032] Furthermore, the 120 acquisition unit records the camera position based on the world coordinate system. The camera position is represented by an external parameter matrix ((R|T)). The camera position can be calculated, for example, using camera calibration.

[0033] Furthermore, the detection unit 120 detects an angle θ between a straight line passing through the camera 11 and a straight line passing through the target object setting point 21. The angle θ is shown below.

[0034] Fig. Figure 8 is a representation showing an example of the angle θ. Fig. Figure 8 shows the angle θ. The detection unit 120 detects the angle θ. The angle θ can be represented as follows. The angle θ is an angle between the plane on which the target object 20 lies when the image is created and a straight line, which represents an image capture direction as the direction in which the image of the target object 20 is captured.

[0035] The 120 acquisition unit records each image, the camera position, the target object alignment point, the target object alignment plane, and the angle θ in the measurement table. The measurement table is shown below.

[0036] Fig. Figure 9 is a representation showing an example of a measurement table. Measurement table 111 is stored in memory unit 110. Measurement table 111 contains elements such as target object identifier (ID), measurement ID, image number, image, camera position, target object adjustment point, target object adjustment plane, angle, and foreground image.

[0037] The Target ID field contains an identifier for the target object. The Measurement ID field contains an identifier for the measurement. The Image Number field contains a number corresponding to the captured image. The Captured Image field contains the captured image. The Camera Position field contains the captured camera position. The Target Setting Point field contains the captured target setting point. The captured target setting point is represented by three-dimensional coordinates in the world coordinate system. The Target Setting Plane field contains the captured target setting plane. The captured plane for target setting can be displayed using printout (1). ax+by+cz+d=0

[0038] As described above, the level at which the target object is set can be represented by expression (1). Therefore, the values ​​of a, b, c, and d are entered in the level for setting the target object. For example, (a, b, c, d) is represented as (0, 0, 1, 0).

[0039] The recorded angle θ is entered under the "Angle" field. A foreground image, which will be described later, is registered under the "Foreground Image" field.

[0040] Furthermore, measurement table 111 may contain information on the lighting conditions and the ID of the camera model.

[0041] The following describes the foreground image generation unit 130.

[0042] The foreground image generation unit 130 generates the foreground image based on the image. In other words, the foreground image generation unit 130 extracts a region of the target object 20 contained in the image and generates the extracted region as the foreground image. The foreground image generation unit 130 can generate the foreground image using conventional technology. Furthermore, the foreground image generation unit 130 can generate the foreground image using the following method.

[0043] Fig. Figure 10 is a block diagram showing the functions of the foreground image generation unit. The foreground image generation unit 130 comprises an area calculation unit 131, a mask image generation unit 132, and a foreground image generation unit 133.

[0044] The area calculation unit 131 calculates the coordinates s of the target object 20 in the image. Specifically, the area calculation unit 131 calculates the coordinates s using expression (2). Furthermore, f represents the image number. The character u f represents the number of pixels in a transverse direction. The symbol v f (R|T) represents the number of pixels in the longitudinal direction. K is the camera's internal parameter. K is represented by a 3 × 3 matrix. (R|T) is an external parameter matrix. (R|T) is represented by a 3 × 4 matrix. The symbol q f stands for a four-dimensional vector (X) f , Y f , Z f , 1), which is obtained by adding 1 to the end of the world coordinates (X f , Y f , Z f ) of the target object setting point 21 in image f is obtained. s(ufvf1)=K(R|t)qf

[0045] The area calculation unit 131 calculates a rectangular area that encloses the target object 20. For example, the area calculation unit 131 calculates the rectangular area by calculating a multitude of points surrounding the coordinates s of the target object 20, using the size of the target object 20 and the world coordinates of the target object setting point 21.

[0046] The following are examples of the image captured by the 120 capture unit. Examples for the rectangular area are also shown.

[0047] Fig. Figure 11 is a representation showing an example (No. 1) of the image. Fig. Figure 11 shows image 40. Image 40 is the image captured by the acquisition unit 120. Image 40 contains the target object 20.

[0048] Fig. 12(A) and Fig. 12(B) are illustrations showing an example (No. 2) of the image. Fig. 12(A) shows the target object setting point 21. Fig. 12(B) shows a rectangular area 41.

[0049] The mask image generation unit 132 creates a mask image using the rectangular area. For example, the mask image generation unit 132 calculates an area representing the target object 20 in pixel units using an algorithm such as GraphCut and produces an image from which the area representing the target object 20 is excluded as a mask image. An example of a mask image is shown below.

[0050] Fig. 13(A) and Fig. 13(B) are illustrations showing an example (No. 3) of the image. Fig. Figure 13(A) shows a mask image 42. The mask image 42 can be represented by a binarized image. The area representing the target object 20 is, for example, labeled 1. An area representing a region other than the target object 20 is labeled 0.

[0051] The foreground image generation unit 133 generates the foreground image using the mask image. For example, the foreground image generation unit 133 generates the foreground image by multiplying the value of each pixel of image 40 by the value of each pixel of mask image 42. Fig. For example, 13(B) shows a foreground image 43.

[0052] The foreground image generation unit 133 enters the foreground image into the measurement table 111.

[0053] The following describes a process carried out by the foreground image generation unit 130 using a flowchart.

[0054] Fig. Figure 14 is a flowchart showing an example of the process performed by the foreground image generation unit.

[0055] (Step S11) The area calculation unit 131 calculates the coordinates s of the target object 20 in the image.

[0056] (Step S12) The area calculation unit 131 calculates the rectangular area.

[0057] (Step S13) The mask image generation unit 132 generates the mask image using the rectangular area.

[0058] (Step S14) The foreground image generation unit 133 generates the foreground image using the mask image. The foreground image shows the target object 20.

[0059] (Step S15) The foreground image generation unit 133 registers the foreground image in the measurement table 111.

[0060] As shown in measurement table 111, an angle is assigned to the foreground image. This angle is also called the foreground image angle. This angle is, for example, angle θ. This angle is the angle between the plane on which the target object 20 lay when the original foreground image was created and the line representing the image acquisition direction, i.e., the direction in which the image of the target object 20 is captured.

[0061] The process from Fig. Procedure 14 can also be performed for all images captured by the capture unit 120.

[0062] The following describes the data generation unit 140.

[0063] The data generation unit 140 overlays the foreground image with the background image. The background image is captured by the acquisition unit 120. The acquisition unit 120, for example, takes the background image from the storage unit 110. Furthermore, the acquisition unit 120 also captures the background image from an external device, for example.

[0064] Point cloud data with three-dimensional coordinates corresponding to each pixel of the background image were assigned to the background image. Therefore, once the background image has been captured, it can be said that the point cloud data has been captured. It is also possible that the capture unit 120 captures the point cloud data at a time different from that of the background image. The three-dimensional coordinates are represented by a camera coordinate system. The camera's coordinate system is described below.

[0065] Fig. Figure 15 is a diagram showing an example of the camera's coordinate system. In the camera coordinate system, the camera position is used as the origin. The camera's image capture direction is the positive direction of the Z-axis.

[0066] The camera position at the time the background image was created was assigned to the background image. Therefore, when the background image was captured, it can be said that the camera position was also captured. It is also possible that the capture unit 120 captured the camera position at a time different from the time the background image was captured. The camera position is the origin of the camera coordinate system and is therefore represented as (0, 0, 0).

[0067] Furthermore, overlay position information can be assigned to the background image, specifying an overlay position at which the foreground image is overlaid on the background image.

[0068] The functions of the data generation unit 140 are described in detail below.

[0069] Fig. Figure 16 is a diagram showing the functions of the data generation unit. The data generation unit 140 comprises a superimposed position determination unit 141, a calculation unit 142, a search unit 143, a selection unit 144, a transformation unit 145, and a superimposed unit 146.

[0070] A process performed by the overlay position determination unit 141 and the calculation unit 142 is described below using a flowchart.

[0071] Fig. Figure 17 is a flowchart showing an example (No. 1) of the process performed by the overlay position determination unit and the computation unit.

[0072] (Step S21) The superposition position determination unit 141 determines the superposition position p Bin the background image. The overlay position p B is a position that corresponds to the target object setting point 21. The overlay position p B is represented by three-dimensional coordinates.

[0073] The superposition position determination unit 141 can determine the superposition position p B randomly determine. Furthermore, the superposition position determination unit 141 can determine the superposition position p. B determine based on the overlay position information.

[0074] (Step S22) The computation unit 142 attempts to define an area (i.e., level P) using the point cloud data. B ) to calculate the superimposed position p B It contains. The calculation unit 142, for example, attempts to calculate the level p. B to be calculated using the point cloud data and the RANSAC algorithm. The plane P B can be represented by expression (1).

[0075] If the plane P B If the calculation is successful, the procedure proceeds to step S23. If the level P B If the calculation is not successful, the procedure proceeds to step S21. In step S21, the superimposition position determination unit 141 determines a new superimposition position.

[0076] (Step S23) The calculation unit 142 calculates an angle θ B between a straight line representing the camera position and the superimposed position p B with the plane P B connects. The angle θ B is also referred to as background image angle.

[0077] In cases where an overlay area has previously been defined in the background image, the following operation can be performed.

[0078] Fig. Figure 18 is a flowchart showing an example (No. 2) of the process performed by the overlay position determination unit and the computation unit.

[0079] (Step S31) The acquisition unit 120 acquires information specifying the overlay area. For example, the acquisition unit 120 retrieves the information about the overlay area from the storage unit 110.

[0080] (Step S32) The superposition position determination unit 141 determines the superposition position p B in the superposition area. For example, the superposition position determination unit 141 determines the superposition position p. B random in the overlapping area.

[0081] (Step S33) The computation unit 142 attempts to determine the area (i.e., the plane P). B ) including the overlay position, using the point cloud data. For example, calculation unit 142 attempts to calculate the plane P B to be calculated using the point cloud data and the RANSAC algorithm.

[0082] If the plane P BIf the calculation is successful, the procedure proceeds to step S34. If the level P B If the calculation is not successful, the procedure proceeds to step S32. In step S32, the superimposition position determination unit 141 determines a new superimposition position.

[0083] (Step S34) The calculation unit 142 calculates the angle θ B between the straight line, which represents the superposition position p B and connects the camera position assigned to the background image, and the plane P B .

[0084] The following describes a process performed by the search unit 143, the selection unit 144, the transformation unit 145 and the overlay unit 146, using a flowchart.

[0085] Fig. Figure 19 is a flowchart showing an example (No. 1) of a superposition process. Fig. Figure 20 is a flowchart showing example (No. 2) of the superposition process.

[0086] (Step S41) The search unit 143 searches in the measurement table 111 for candidates for the foreground image based on the angle θ. B as foreground image candidates. Specifically, search unit 143 searches for the foreground image candidates by selecting the angle element in measurement table 111 based on the angle θ. B searched. In addition, it is possible that the search unit 143 searches as foreground image candidates for foreground images that correspond to (i.e. are associated with) an angle θ that meets the condition "within a range of the angle θ". B ± 1°“ is satisfied. In this way, one or more foreground image candidates are sought, each of which has an angle equal to or close to the angle θ. B is assigned.

[0087] The acquisition unit 120 acquires one or more foreground image candidates (i.e., foreground images) by searching the search unit 143. It is also possible for the acquisition unit 120 to acquire the foreground image candidates from an external device.

[0088] (Step S42) Selection unit 144 selects one foreground image candidate from one or more foreground image candidates.

[0089] Even if the angles are the same or close to each other, the appearance of the individual foreground image candidates changes when the shooting direction changes. A specific case in which the appearance varies is illustrated below.

[0090] Fig. 21(A) and Fig. 21(B) are illustrations that show an example of the case where the appearance varies. Fig. Figure 21(A) shows an image 50 containing the target object 20. Fig. Figure 21(B) shows image 51 containing the target object 20.

[0091] The angle θ corresponding to image 50 and the angle θ corresponding to image 51 are equal or very close to each other. Fig. 21(A) and Fig. However, Figure 21(B) shows a case in which the image acquisition direction of the target object 20 varies. As in Fig. 21(A) and Fig. As shown in Figure 21(B), the appearance changes even when the angles are the same or close to each other, if the image capture direction changes. Therefore, the transformation unit 145 performs a transformation process.

[0092] First, the selected foreground image candidate is designated as image A. The camera position is represented by the external parameter matrix ((R|T)). The camera position corresponding to image A is designated as camera position R. A Assuming the camera position is R. AThis corresponds to “R” in the camera position “(R |T)”, which corresponds to image A. The target object setting point that corresponds to image A is called target object setting point p. A Assuming the target object setting plane corresponding to image A is designated as target object setting plane P. A assumed. Furthermore, the superimposition position p B and the level P B , which corresponds to the background image, has already been captured. A camera position corresponding to the background image R B is an unknown value.

[0093] (Step S43) Transformation unit 145 calculates fundamental vectors of an object position coordinate system C in image A. An example of the object position coordinate system C is shown below.

[0094] Fig. Figure 22 is a diagram showing an example of a coordinate system for setting the object position. In the coordinate system of the object setting position C, a vector connecting the camera and the target object setting point is used to define the Z-axis. The transformation unit 145 performs a projective transformation so that the camera position R A In the coordinate system, object position C is equal to camera position R. B in the coordinate system of object position C.

[0095] The calculation of the fundamental vectors of the position coordinate system C of the object in image A is described in detail below. Here, the world coordinates of the camera position are defined as O. A Assuming the world coordinates of the target object's setting point p. A are considered X A Assume the world coordinates of an intersection point between the target object setting point plane P Aand a normal vector passing through O A runs, are called Y A assumed.

[0096] The direction of the X-axis of the fundamental vectors of the coordinate system of the object position C is represented by expression (3). xA→=[ex1→ex2→ex3→]=OAXA→×OAYA→|OAXA→×OAYA→|

[0097] The direction of the Y-axis of the fundamental vectors of the coordinate system of the object position C is represented by expression (4). yA=[ey1→ey2→ey3→]=xA→×zA→|xA→×zA→|

[0098] The direction of the Z-axis of the fundamental vectors of the object's position coordinate system C is represented by expression (5). zA→=[ez1→ez2→ez3→]=OAXA→OAXA→

[0099] (Step S44) Transformation unit 145 calculates fundamental vectors of the object position coordinate system C in the background image. The camera position in the camera coordinate system is represented as O.B Assume the world coordinates of the superposition position p. B are considered X B Assume the world coordinates of an intersection point between the plane P B and a normal vector passing through O B runs, are called Y B assumed.

[0100] The direction of the X-axis of the fundamental vectors of the coordinate system of the object position C is represented by expression (6). xB→=OBXB→×OBYB→|OBXB→×OBYB→|

[0101] The direction of the Y-axis of the fundamental vectors of the coordinate system of the object position C is represented by expression (7). yB→==xB→×zB→|xB→×zB→|

[0102] The direction of the Z-axis of the fundamental vectors of the coordinate system of the object position C is represented by expression (8). e.g.→=OBXB→|OBXB→|

[0103] A matrix resulting from the vertical connection of the basis vectors of the image A is called T. A Assumed. T A is represented by expression (9). TA=xA→+yA→+zA→

[0104] As T B A matrix is ​​assumed that results from the vertical connection of the basic vectors of the background image. T B is represented by expression (10). TB=xB→+yB→+zB→

[0105] The camera position R A1 The position of image A in the coordinate system of object position C is represented by expression (11). RA1=TA−1RA

[0106] The camera position R B1 The background image in the coordinate system of object position C is represented by expression (12). RB1=TB−1

[0107] A camera position R BThe background image in the world coordinate system is represented by expression (13). Furthermore, the camera position R B a rotation matrix. RB=TARB1

[0108] A transformation matrix T AB The camera's position is represented by expression (14). TAB=RBRA−1

[0109] (Step S45) The transformation unit 145 calculates a homography matrix H AB using expression (15). K is, incidentally, an internal parameter matrix of the camera. HAB=KTABK

[0110] (Step S46) Transformation unit 145 performs the projective transformation on image A using expression (16). The projective transformation in expression (16) is also known as the homography transformation. Furthermore, x and y are the pixel coordinates of image A, while x new and y neware the pixel coordinates of image A after the projective transformation. (xnewynew1)=HAB(xy1)

[0111] As described above, the transformation unit 145 performs the projective transformation of image A in the image acquisition direction in the background image. In other words, the transformation unit 145 performs the projective transformation of image A in the image acquisition direction of the camera that created the background image.

[0112] If the image capture direction in the background image and the image capture direction in image A are identical, image A will not be projectively transformed, even if steps S43 to S46 are performed.

[0113] Fig. 23(A) and Fig. 23(B) are illustrations (No. 1) to explain the projective transformation. Fig. 23(A) shows image 60 as image A. Fig. 23(B) shows image 61 as a background image.

[0114] The angle θ corresponding to image 60, and the angle θ B The two images corresponding to image 61 are identical. However, the image acquisition direction corresponding to image 60 and the image acquisition direction corresponding to image 61 differ by 90 degrees. This results in a different appearance. Therefore, transformation unit 145 performs the projective transformation of image A.

[0115] Fig. Figure 24 is a representation (No. 2) to explain the projective transformation. Fig. Figure 24 shows image 62 as image A after the projective transformation. As described above, the projective transformation makes the appearance of the foreground image identical to the appearance of the background image.

[0116] After step S46, the procedure proceeds to step S51.

[0117] (Step S51) Transformation unit 145 assesses whether the distance between the camera and the target object adjustment plane differs between image A and the background image. In this case, the original foreground image, as image A, is an image generated by an RGB-D camera. Therefore, a distance has been assigned to each pixel of image A (i.e., the foreground image). Additionally, point cloud data corresponding to each pixel of the background image has been assigned to the background image. Thus, the distance is determined by the Z-coordinate represented by the point cloud data.

[0118] If the distance between image A and the background image is different, this means that the scale between image A and the background image is different. If the distance between image A and the background image is different, proceed to step S52. If the distance in image A and the background image is the same, proceed to step S53.

[0119] (Step S52) The transformation unit 145 performs a scale transformation of image A.

[0120] The scaling transformation is described in detail below. First, the distance in image A (i.e., the distance between the camera position and an intersection point between the target object and the setting point plane P) is calculated. A and a vector representing the image capture direction) with d A Assume the distance in the background image (i.e., the distance between the camera position and an intersection point between the plane P). Band a vector representing the image capture direction) is combined with d B Assumption. Furthermore, the following description assumes that image A is subject to projective transformation.

[0121] Transformation unit 145 performs the scale transformation of image A using the affine transformation. Specifically, transformation unit 145 performs the scale transformation of image A using expression (17). (xscale_newyscale_new1)=(dAdB000dAdB0001)(xnewynew1)

[0122] The following is an example of image A after the scale transformation.

[0123] Fig. Figure 25 is a representation showing an example of an image after the scale transformation. The transformation unit 145 performs the scale transformation of image 62 as image A. This makes the target object 20 contained in image 62 smaller.

[0124] The above description refers to the case where the projective transformation is applied to image A. If the image acquisition direction in image A and the image acquisition direction in the background image are the same, the transformation unit 145 does not perform a projective transformation on image A. Furthermore, the case where the scale transformation is performed on image A after the projective transformation was described above. It is also possible for the transformation unit 145 to perform the scale transformation on image A that has not undergone the projective transformation.

[0125] (Step S53) Selection unit 144 assesses whether all foreground image candidates have been selected. If all foreground image candidates have been selected, the process continues with step S54. If not all foreground image candidates have been selected, the process continues with step S42.

[0126] (Step S54) Selection unit 144 selects from one or more foreground image candidates a foreground image candidate that is optimal as the foreground image to be superimposed. In other words, selection unit 144 selects from the foreground image candidates an image that minimizes the change in viewing angle caused by the image transformation.

[0127] Here, an index d(P, Q) is introduced, representing the degree of similarity between a rotation matrix P and a rotation matrix Q. A smaller value of the index d(P, Q) means that the rotation matrix P and the rotation matrix Q are more similar to each other. A smaller value of the index d(P, Q) indicates that the change in viewing angle is smaller, independent of any change in camera position. The index d(P, Q) can be represented using expression (18). d(P,Q)=∫|Px−Qx|2dx x ∈ POINTS ON THE UNIT SPHERE

[0128] The index d(P, Q) is calculated ad hoc by randomly selecting points on a unit sphere and calculating the distance between the coordinates obtained by the transformation with the rotation matrix P and the coordinates obtained by the transformation with the rotation matrix Q with respect to each of the points.

[0129] When selecting the optimal foreground image candidate, “P” in the index d(P, Q) is determined by the camera position R. A Furthermore, “Q” in the index d(P, Q) is replaced by the camera position R. B replaced.

[0130] If there are N sheets of foreground image candidates, the selection unit 144 selects an image i as the optimal foreground image candidate using expression (19). i=argminj∈(1,…,N)d(RA(j),RB)

[0131] (Step S55) The overlay unit 146 overlays the selected image i (i.e., the foreground image) with the background image. Specifically, the overlay unit 146 places the foreground image at the overlay position p. B of the background image.

[0132] The above description was given assuming that the projective transformation and the scale transformation are performed on image A, for the sake of clarity. In reality, however, the projective transformation and the scale transformation are performed on the foreground image, which displays the target object 20 in image A.

[0133] An example of a flow from projective transformation to superposition is briefly described below.

[0134] Fig. Figure 26 is a representation showing the transition from projective transformation to superposition. Fig.Figure 26 shows a foreground image 70. The data generation unit 140 performs the projective transformation of the foreground image 70. The data generation unit 140 performs the scale transformation of the foreground image 70. In other words, the data generation unit 140 converts the scale of the foreground image 70 to the scale of a background image 71. The data generation unit 140 overlays the foreground image 70 with the background image 71. In this way, the combined image is generated. The combined image is used as training data. Furthermore, the information recorded in the measurement table 111 can be assigned to the combined image. Assigning the information to the combined image makes it possible to narrow down the training data. In addition, assigning the information to the combined image allows for the extraction of training data that corrects a state imbalance if one has occurred.Furthermore, information about the correct answer can be assigned to the combined image.

[0135] The information processing unit 100 overlays the background image with a foreground image corresponding to an angle θ that is equal to or close to the angle θ. B of the background image. Therefore, the combined image created by overlaying the foreground image with the background image is displayed as realistic data. Thus, the Information Processing Unit 100 is capable of generating realistic data.

[0136] If the image capture direction in the foreground image and the image capture direction in the background image differ, the information processing unit 100 performs a projective transformation on the foreground image. This aligns the direction of the foreground image with the image capture direction of the background image. The information processing unit 100 then overlays the foreground image with the background image after the projective transformation. Therefore, the combined image resulting from the overlay of the foreground image and background image is displayed as more realistic data. This enables the information processing unit 100 to generate more realistic data.

[0137] If the scale of the foreground image and the scale of the background image differ, the information processing unit 100 performs a scale transformation on the foreground image. This transforms the foreground image to the same scale as the background image. The information processing unit 100 then overlays the foreground image with the background image after the scale transformation. Therefore, the combined image created by overlaying the foreground image with the background image is displayed as more realistic data. This enables the information processing unit 100 to generate more realistic data.

[0138] The information processing unit 100 selects an optimal foreground image candidate from a large number of foreground image candidates to be superimposed. The information processing unit 100 then superimposes this selected foreground image candidate with the background image. Therefore, the combined image resulting from the superimposition of the foreground image and the background image is considered to be more optimal data. The information processing unit 100 is thus capable of generating more optimal data. REFERENCE MARK LIST

[0139] 10: Robot arm, 11: Camera, 20: Target object, 21: Target object setting point, 30: Belt conveyor, 31: Object, 32: Camera, 40: Image, 41: Rectangular area, 42: Mask image, 43: Foreground image, 50: Image, 51: Image, 60: Image, 61: Image, 62: Image, 70: Foreground image, 71: Background image, 100: Information processing device, 101: Processor, 102: Volatile memory device, 103: Non-volatile memory device, 104: Input device, 105: Display device, 110: Storage unit, 111: Measurement table, 120: Acquisition unit, 130: Foreground image generation unit, 131: Area calculation unit, 132: Mask image generation unit, 133: Foreground image generation unit, 140: Data generation unit, 141: Overlay position determination unit, 142: Calculation unit, 143: Search unit, 144: Selection unit, 145: Transformation unit, 146: Overlay unit

Claims

[1] Information processing equipment (100), comprising: a capture unit (120) that captures a foreground image to which a foreground image angle is assigned and which displays a target object (20), a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position at the time the background image is generated; an overlay position determination unit (141) that determines an overlay position in the background image; a computation unit (142) that computes a plane as an area containing the overlay position using the point cloud data and computes a background image angle as an angle between a straight line connecting the camera position and the overlay position and the computed plane; a transformation unit (145) that performs a projective transformation on the foreground image in the image acquisition direction in the background image such that a camera position corresponding to the foreground image in a world coordinate system becomes equivalent to a camera position corresponding to the background image in the world coordinate system; and a superimposition unit (146) which overlays the foreground image onto the background image to create a combined image, wherein the foreground image angle is an angle between a plane on which the target object (20) was positioned when an original foreground image was produced and a straight line representing an image acquisition direction as a direction in which the image of the target object (20) is acquired, and is an angle which is equal to or close to the background image angle. [2] Information processing device according to claim 1, wherein the transformation unit (145) performs a scale transformation of the foreground image to a scale of the background image. [3] Information processing device according to claim 1 or 2, further comprising a selection unit (144), wherein the acquisition unit (120) acquires a plurality of foreground image candidates, each assigned an angle that is equal to or close to the background image angle, the selection unit (144) selects a foreground image candidate which is optimal from the multitude of foreground image candidates as the foreground image to be superimposed, and The foreground image, which is overlaid on the background image, is the selected foreground image candidate. [4] Generation process performed by an information processing device (100), the generation process comprising: Capturing a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position at the time the background image was created; Determining an overlay position in the background image; Calculating a plane as an area that includes the overlay position, using the point cloud data; Calculating a background image angle as an angle between a straight line connecting the camera position and the overlay position, and the calculated plane; Capturing a foreground image to which a foreground image angle is assigned that is equal to or close to the background image angle and displays a target object (20); Performing a projective transformation on the foreground image in the image capture direction in the background image such that a camera position corresponding to the foreground image in a world coordinate system becomes equivalent to a camera position corresponding to the background image in the world coordinate system; and Overlaying the foreground image with the background image to create a combined image, where the foreground image angle is an angle between a plane on which the target object (20) was positioned when an original image of the foreground image was produced, and a straight line representing an image acquisition direction as a direction in which the image of the target object (20) is captured. [5] Generation program that causes an information processing unit (100) to perform an operation of: Capturing a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position at the time the background image was generated; Determining an overlay position in the background image; Calculating a plane as an area that includes the overlay position, using the point cloud data; Calculating a background image angle as an angle between a straight line connecting the camera position and the overlay position, and the calculated plane; Capturing a foreground image to which a foreground image angle is assigned that is equal to or close to the background image angle and displays a target object (20); Performing a projective transformation on the foreground image in the image capture direction in the background image such that a camera position corresponding to the foreground image in a world coordinate system becomes equivalent to a camera position corresponding to the background image in the world coordinate system; and Overlaying the foreground image with the background image to create a combined image, where the foreground image angle is an angle between a plane on which the target object (20) was positioned when an original image of the foreground image was produced, and a straight line representing an image acquisition direction as a direction in which the image of the target object (20) is captured.

Images