Information processing device, generation method, and generation program

Abstract

An information processing device (100) is provided with: an acquisition unit (120) that acquires a foreground image that corresponds to an angle (theta) and represents an object, a background image, point cloud data that represents three-dimensional coordinates corresponding to each pixel of the background image, and a camera position at the time of generating the background image; an overlap position determination unit (141) that determines an overlap position from the background image; a calculation unit (142) that uses the point cloud data to calculate a plane that is a region including the overlapping position, and calculates an angle [theta] B that is an angle between a straight line connecting the camera position and the overlapping position and the calculated plane; and a superimposing unit (146) that superimposes the foreground image on the background image. The angle [theta] is an angle between a plane on which the object is provided when the original image of the foreground image is generated and a straight line indicating an imaging direction, which is the direction in which the object is imaged, and is the same as or similar to the angle [theta] B.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, a generation method, and a generation program. Background Art

[0002] It is known to use a learned model for object recognition. A large number of images are required in the learning stage of the learned model. It is possible to prepare a large number of images containing common objects such as people, cars, animals, etc. For example, if open source is used, it is possible to prepare a large number of images containing common objects. However, it is difficult to prepare a large number of images of specific objects such as specific cars, equipment, products, etc. In addition, when the posture of the object changes, it is difficult to prepare a large number of objects in various postures. Therefore, a technology for generating images is proposed (see patent document 1). The information processing device of patent document 1 obtains three-dimensional information containing multiple images, selects a recognition object from the three-dimensional information, and cuts out images of the range corresponding to the selected recognition object, that is, the recognition object range, from the multiple images, thereby generating multiple foreground images based on the multiple images, and synthesizing each foreground image in the multiple foreground images with each background image in the multiple background images, thereby generating multiple synthesized images.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2020 / 152763 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, when a foreground image is superimposed on a background image, realistic data may not be generated because no constraints are set. For example, an image is generated in which an object is arranged at an unrealistic angle. When such an image is used as learning data, the recognition accuracy of the learned model decreases.

[0008] The purpose of the present disclosure is to generate realistic data.

[0009] Means used to solve problems

[0010] An information processing device according to one embodiment of the present disclosure is provided. The information processing device comprises: an acquisition unit that acquires a foreground image corresponding to a foreground image angle and representing an object, a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position when generating the background image; an overlapping position determination unit that determines an overlapping position from the background image; a calculation unit that uses the point cloud data to calculate a plane as an area including the overlapping position, and calculates a background image angle, which is an angle between a straight line connecting the camera position and the overlapping position and the calculated plane; and an overlapping unit that overlaps the foreground image on the background image. The foreground image angle is an angle between a plane on which the object is set when generating an original image of the foreground image and a straight line representing a direction in which the object is photographed, i.e., a shooting direction, and is an angle that is the same as or similar to the background image angle.

[0011] Effects of the Invention

[0012] According to the present disclosure, realistic data can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a diagram showing hardware included in the information processing device.

[0014] Figure 2 is a block diagram showing the functions of the information processing device.

[0015] Figure 3 This is a diagram showing an example (part 1) of a method of photographing an object.

[0016] Figure 4 This is a diagram showing an example (part 2) of a method of photographing an object.

[0017] Figure 5 This is a diagram showing an example (part 3) of a method of photographing an object.

[0018] Figure 6 An example is shown in which an object moves in a fixed direction.

[0019] Figure 7 This is a diagram showing an example (part 4) of a method of photographing an object.

[0020] Figure 8 : is a diagram showing an example of the angle θ.

[0021] Fig. 9 It is a figure which shows an example of a measurement table.

[0022] Fig.10 It is a block diagram showing the functions of the foreground image generating unit.

[0023] Fig.11 It is a figure which shows the example of an image (part 1).

[0024] Fig.12 (A) and (B) are diagrams showing examples of images (part 2).

[0025] Fig.13 (A) and (B) are diagrams showing examples of images (part 3).

[0026] Fig.14 : is a flowchart showing an example of processing performed by the foreground image generating unit.

[0027] Fig.15 is a diagram showing an example of a camera coordinate system.

[0028] Fig.16 It is a diagram showing the function of the data generation unit.

[0029] Fig.17 This is a flowchart showing an example (part 1) of processing executed by the superimposition position determination unit and the calculation unit.

[0030] Fig.18 This is a flowchart showing an example (part 2) of processing executed by the superimposition position determination unit and the calculation unit.

[0031] Fig.19 1 is a flowchart showing an example of overlapping processing.

[0032] Fig. 20 This is a flowchart showing an example of overlapping processing (part 2).

[0033] Fig.21 (A) and (B) are diagrams showing examples in different visual conditions.

[0034] Fig. 22 : is a diagram showing an example of an object installation position coordinate system.

[0035] Fig.23 (A) and (B) are diagrams (part 1) for explaining projection transformation.

[0036] Fig.24 This is a diagram for explaining projection transformation (part 2).

[0037] Fig.25 : is a figure which shows the example of the image after the scale conversion.

[0038] Fig.26 It is a diagram showing the flow from projection conversion to superposition. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments will be described with reference to the drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present disclosure.

[0040] Implementation method.

[0041] Figure 1 1 is a diagram showing hardware included in an information processing device. The information processing device 100 is a device that executes the generation method. For example, the information processing device 100 is a server, a PC (Personal Computer), a smartphone, or the like.

[0042] The information processing device 100 includes a processor 101, a volatile storage device 102, a nonvolatile storage device 103, an input device 104, and a display device 105. The input device 104 and the display device 105 may be provided outside the information processing device 100.

[0043] The processor 101 controls the entire information processing device 100. For example, the processor 101 is a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), etc. The processor 101 may also be a multiprocessor. In addition, the information processing device 100 may also have a processing circuit.

[0044] The volatile storage device 102 is a main storage device of the information processing device 100. For example, the volatile storage device 102 is a RAM (Random Access Memory). The nonvolatile storage device 103 is an auxiliary storage device of the information processing device 100. For example, the nonvolatile storage device 103 is a HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0045] The input device 104 is a keyboard, a touch panel, etc. The display device 105 is a display.

[0046] Next, the functions of the information processing device 100 will be described.

[0047] Figure 2 The information processing device 100 includes a storage unit 110 , an acquisition unit 120 , a foreground image generation unit 130 , and a data generation unit 140 .

[0048] The storage unit 110 may be realized as a storage area reserved in the volatile storage device 102 or the nonvolatile storage device 103 .

[0049] Part or all of the acquisition unit 120, the foreground image generation unit 130, and the data generation unit 140 may also be implemented by a processing circuit. In addition, part or all of the acquisition unit 120, the foreground image generation unit 130, and the data generation unit 140 may also be implemented as a module of a program executed by the processor 101. For example, the program executed by the processor 101 is also referred to as a generation program. For example, the generation program is recorded in a recording medium.

[0050] The storage unit 110 stores various information. For example, the storage unit 110 stores a measurement table. The measurement table will be described later.

[0051] The acquisition unit 120 acquires an image including the object 20. For example, the acquisition unit 120 acquires the image from a camera. For example, the camera is an RGB (Red Green Blue)-D (Depth) camera. In addition, when the camera is an RGB camera, the acquisition unit 120 may also acquire the image from a combination of a sensor capable of measuring the distance between the RGB camera and the object 20 (for example, an IMU (Inertial Measurement Unit) sensor, an infrared sensor, a LiDAR (Light Detection And Ranging)) and an RGB camera.

[0052] In addition, the acquisition unit 120 may acquire an image including the object 20 from an external device. For example, the external device is a cloud server. In addition, the external device is omitted in the figure.

[0053] Here, an example of a method of photographing an object will be described.

[0054] Figure 3 This is a diagram showing an example (part 1) of a method of photographing an object. Figure 3 A robot arm 10 is shown. A camera 11 is mounted at the front end of the robot arm 10.

[0055] For example, the installation position of the robot arm 10 is handled as the origin, and the image generated by the camera 11 is processed using the world coordinate system of the origin.

[0056] Figure 4 This is a diagram showing an example (part 2) of a method of photographing an object. Figure 4 An object 20 is shown. The object 20 is placed on a plane where the robot arm 10 is placed. Figure 4The object setting point 21 is shown. The object setting point 21 is a point in contact with the plane and is the centroid or center of the object 20. Therefore, the Z coordinate of the object setting point 21 is 0. In addition, the object setting point 21 can also be determined before shooting. In addition, the object setting point 21 can also be set by adjusting so that the center point of the camera 11 coincides with the center point of the object 20.

[0057] For example, the internal parameters of the camera 11 can also be obtained through camera calibration.

[0058] Next, two imaging methods are described in detail. First, the first imaging method is described.

[0059] Figure 5 2 is a diagram showing an example (part 3) of a method of photographing an object. The robot arm 10 photographs the object 20 from various angles while operating. In addition, the photographing timing, the photographing range, etc. may be changed as appropriate.

[0060] Next, the second photographing method is described. In the second photographing method, photographing is performed using the following method.

[0061] Figure 6 An example is shown in which an object moves in a fixed direction. Figure 6 Belt conveyor 30 is shown. Object 31 is placed on belt conveyor 30. Object 31 moves in a fixed direction. Camera 32 captures the object 31 moving in parallel. This capturing method is used to capture object 20. Specifically, Figure 7 Provide explanation.

[0062] Figure 7 2 is a diagram showing an example (part 4) of a method of photographing an object. The robot arm 10 photographs the object 20 while performing parallel movement.

[0063] By capturing the image of the object 20 in this manner, the information processing device 100 can obtain an image including the object 20. In addition, the method of capturing the image of the object 20 may be a method other than the above-described method.

[0064] In addition, the acquisition unit 120 acquires the position and posture of the camera based on the world coordinate system. The position and posture of the camera is represented by an external parameter matrix ((R|T)). For example, the position and posture of the camera may be calculated by using camera calibration.

[0065] Furthermore, the acquisition unit 120 acquires an angle θ between a straight line passing through the camera 11 and a straight line passing through the object setting point 21. The angle θ is shown.

[0066] Figure 8 : is a diagram showing an example of the angle θ. Figure 8The angle θ is shown. The acquisition unit 120 acquires the angle θ. The angle θ can also be expressed as follows. The angle θ is the angle between the plane on which the object 20 is placed when the image is generated and the straight line indicating the direction in which the object 20 is photographed, that is, the photographing direction.

[0067] The acquisition unit 120 registers the image, the camera position and posture, the object installation point, the object installation plane, and the angle θ in the measurement table.

[0068] Fig. 9 1 is a diagram showing an example of a measurement table. The measurement table 111 is stored in the storage unit 110. The measurement table 111 has items of object ID (Identifier), measurement ID, frame number, image, camera position and posture, object setting point, object setting plane, angle, and foreground image.

[0069] The object ID item has an identifier of the object registered. The measurement ID item has an identifier of the measurement registered. The frame number item has a number corresponding to the acquired image registered. The image item has an acquired image registered. The camera position and posture item has an acquired camera position and posture registered. The object setting point item has an acquired object setting point registered. In addition, the acquired object setting point is represented by a three-dimensional coordinate in the world coordinate system. The object setting plane item has an acquired object setting plane registered. In addition, the acquired object setting plane can be represented by formula (1).

[0070] [Formula 1]

[0071] ax+by+cz+d=0 …(1)

[0072] Thus, the acquired object setting plane can be expressed by equation (1). Therefore, the values ​​of a, b, c, and d are registered in the item of the object setting plane. For example, (a, b, c, d) is expressed by (0, 0, 1, 0).

[0073] The acquired angle θ is registered in the item of angle. The foreground image, which will be described later, is registered in the item of foreground image.

[0074] Furthermore, the measurement table 111 may include items of lighting conditions and camera model ID.

[0075] Next, the foreground image generating unit 130 will be described.

[0076] The foreground image generating unit 130 generates a foreground image based on the image. In other words, the foreground image generating unit 130 extracts a region of the object 20 included in the image and generates the extracted region as a foreground image. The foreground image generating unit 130 may also generate a foreground image using existing technology. In addition, the foreground image generating unit 130 may also generate a foreground image using the following method.

[0077] Fig.10 2 is a block diagram showing the functions of the foreground image generating unit 130. The foreground image generating unit 130 includes a region calculating unit 131, a mask image generating unit 132, and a foreground image generating unit 133.

[0078] The region calculation unit 131 calculates the coordinates s of the object 20 in the image. Specifically, the region calculation unit 131 calculates the coordinates s using equation (2). In addition, f is a frame number. f is the number of pixels in the horizontal direction. f is the number of pixels in the vertical direction. K is the internal parameter of the camera. K is represented by a 3×3 matrix. (R|T) is the external parameter matrix. (R|T) is represented by a 3×4 matrix. f is the world coordinate (X f , Y f , Z f ) is a four-dimensional vector (X f , Y f , Z f , 1).

[0079] [Formula 2]

[0080]

[0081] The area calculation unit 131 calculates a rectangular area including the object 20. For example, the area calculation unit 131 calculates a plurality of points surrounding the coordinates s of the object 20 using the size of the object 20 and the world coordinates of the object setting point 21, thereby calculating the rectangular area.

[0082] The following is an example of an image acquired by the acquisition unit 120. In addition, an example of the rectangular region is also shown.

[0083] Fig.11 It is a figure which shows the example of an image (part 1). Fig.11 An image 40 is shown.

[0084] The image 40 is an image acquired by the acquisition unit 120 . The image 40 includes the object 20 .

[0085] Fig.12 (A) and (B) are diagrams showing examples of images (part 2). Fig.12(A) shows the object setting point 21. Fig.12 (B) shows a rectangular area 41 .

[0086] The mask image generation unit 132 generates a mask image using the rectangular area. For example, the mask image generation unit 132 calculates the area representing the object 20 in pixel units using an algorithm such as GraphCut, and generates an image as a mask image after removing the area representing the object 20. The mask image is shown below.

[0087] Fig.13 (A) and (B) are diagrams showing examples of images (part 3). Fig.13 (A) shows a mask image 42. The mask image 42 can be represented by a binary image. For example, a region representing the object 20 is represented by 1. A region representing a region other than the object 20 is represented by 0.

[0088] The foreground image generation unit 133 generates a foreground image using the mask image. For example, the foreground image generation unit 133 generates a foreground image by multiplying the value of each pixel of the mask image 42 by the value of each pixel of the image 40. For example, Fig.13 (B) shows a foreground image 43 .

[0089] The foreground image generating unit 133 registers the foreground image in the measurement table 111 .

[0090] Next, the processing executed by the foreground image generating unit 130 will be described using a flowchart.

[0091] Fig.14 : is a flowchart showing an example of processing performed by the foreground image generating unit.

[0092] (Step S11 ) The area calculation unit 131 calculates the coordinates s of the object 20 in the image.

[0093] (Step S12 ) The area calculation unit 131 calculates a rectangular area.

[0094] (Step S13) The mask image generating unit 132 generates a mask image using the rectangular region.

[0095] (Step S14 ) The foreground image generating unit 133 generates a foreground image using the mask image. The foreground image represents the object 20 .

[0096] (Step S15 ) The foreground image generating unit 133 registers the foreground image in the measurement table 111 .

[0097] As shown in the measurement table 111, the angle is associated with the foreground image. This angle is also called the foreground image angle. For example, this angle is the angle θ. This angle is the angle between the plane on which the object 20 is set when the original image of the foreground image is generated and the straight line indicating the direction of shooting the object 20, that is, the shooting direction.

[0098] In addition, all images acquired by the acquisition unit 120 may be processed. Fig.14 processing.

[0099] Next, the data generating unit 140 will be described.

[0100] The data generating unit 140 superimposes the foreground image on the background image. The background image is acquired by the acquiring unit 120. For example, the acquiring unit 120 acquires the background image from the storage unit 110. In addition, for example, the acquiring unit 120 acquires the background image from an external device.

[0101] The point cloud data representing the three-dimensional coordinates corresponding to each pixel of the background image corresponds to the background image. Therefore, when the background image is obtained, it can be expressed as the point cloud data is obtained. In addition, the acquisition unit 120 can also obtain the point cloud data at a time different from the time of obtaining the background image. The three-dimensional coordinates are represented by the camera coordinate system. The camera coordinate system is explained.

[0102] Fig.15 : is a diagram showing an example of a camera coordinate system. In the camera coordinate system, the camera position is the origin. The shooting direction of the camera is the positive direction of the Z axis.

[0103] The camera position when the background image is generated corresponds to the background image. Therefore, when the background image is acquired, it can be expressed as the camera position being acquired. In addition, the acquisition unit 120 can also acquire the camera position at a different time from the acquisition time of the background image. Since the camera position is the origin of the camera coordinate system, it is represented by (0, 0, 0).

[0104] Furthermore, superimposed position information where the foreground image is superimposed on the background image may be associated with the background image.

[0105] Next, the function of the data generating unit 140 will be described in detail.

[0106] Fig.16 140 is a diagram showing the functions of the data generation unit. The data generation unit 140 includes a superimposed position determination unit 141 , a calculation unit 142 , a search unit 143 , a selection unit 144 , a conversion unit 145 , and a superimposition unit 146 .

[0107] The processing executed by the superimposed position determination unit 141 and the calculation unit 142 will be described using a flowchart.

[0108] Fig.17 This is a flowchart showing an example (part 1) of processing executed by the superimposition position determination unit and the calculation unit.

[0109] (Step S21) The superimposition position determination unit 141 determines a superimposition position p from the background image. B . Overlap position p B This is the position corresponding to the object setting point 21. The overlapping position p B The superimposition position determination unit 141 may also randomly determine the superimposition position p B In addition, the superimposed position determination unit 141 may also determine the superimposed position p based on the superimposed position information. B .

[0110] (Step S22) The calculation unit 142 uses the point cloud data to try to calculate the overlap position p. B The area (i.e., plane P B ). For example, the calculation unit 142 uses the point cloud data and the RANSAC algorithm to try to calculate the plane P B In addition, plane P B It can be expressed by formula (1).

[0111] In calculating the plane P B If the plane P is not calculated, the process proceeds to step S23. B In the case of , the process proceeds to step S21. In this step S21, the superimposition position determination unit 141 determines a new superimposition position.

[0112] (Step S23) The calculation unit 142 calculates the connection camera position and the overlapping position p B The line and plane P B The angle θ between B Angle θ B Also called background image angle.

[0113] Furthermore, when the overlapping range is predetermined in the background image, the following processing may be performed.

[0114] Fig.18 This is a flowchart showing an example (part 2) of processing executed by the superimposition position determination unit and the calculation unit.

[0115] (Step S31 ) The acquisition unit 120 acquires information indicating the overlapping range. For example, the acquisition unit 120 acquires information indicating the overlapping range from the storage unit 110 .

[0116] (Step S32) The superimposition position determination unit 141 determines the superimposition position p from the superimposition range. B For example, the superposition position determination unit 141 randomly determines the superposition position p from the superposition range.B .

[0117] (Step S33) The calculation unit 142 uses the point cloud data to try to calculate the area including the overlapping position (ie, the plane P B ). For example, the calculation unit 142 uses the point cloud data and the RANSAC algorithm to try to calculate the plane P B .

[0118] In calculating the plane P B If the plane P is not calculated, the process proceeds to step S34. B In the case of , the process proceeds to step S32. In this step S32, the superimposition position determination unit 141 determines a new superimposition position.

[0119] (Step S34) The calculation unit 142 calculates the following straight line and plane P B The angle θ between B , which connects the camera position corresponding to the background image and the overlapping position p B .

[0120] Next, the processing executed by the search unit 143 , the selection unit 144 , the conversion unit 145 , and the superposition unit 146 will be described using a flowchart.

[0121] Fig.19 1 is a flowchart showing an example of overlapping processing. Fig. 20 This is a flowchart showing an example of overlapping processing (part 2).

[0122] (Step S41) The search unit 143 performs the search based on the angle θ. B , the foreground image candidate is retrieved from the measurement table 111 as the foreground image candidate. Specifically, the retrieval unit 143 retrieves the foreground image candidate based on the angle θ B , the angle item of the measurement table 111 is searched, thereby searching for the foreground image candidate. In addition, for example, the search unit 143 may also search for the items related to the "angle θ B ±1°” as the foreground image candidate. Thus, the foreground image related to the angle θ is retrieved. B One or more foreground image candidates of the same or similar angle.

[0123] The acquisition unit 120 acquires one or more foreground image candidates (ie, foreground images) through the search by the search unit 143. Alternatively, the acquisition unit 120 may acquire one or more foreground image candidates from an external device.

[0124] (Step S42 ) The selection unit 144 selects one foreground image candidate from one or more foreground image candidates.

[0125] Here, when the angles are the same or similar but the shooting directions are different, the visual conditions are different. Specifically, the case where the visual conditions are different is shown.

[0126] Fig.21 (A) and (B) are diagrams showing examples in different visual conditions. Fig.21 (A) shows an image 50 including the object 20 . Fig.21 (B) shows an image 51 including the object 20 .

[0127] The angle θ corresponding to the image 50 is the same as or similar to the angle θ corresponding to the image 51. However, Fig.21 (A) and Fig.21 (B) shows a case where the shooting direction of the object 20 is different. Fig.21 (A) and Fig.21 As shown in (B), when the angles are the same or similar but the shooting directions are the same, the visual conditions are different. Therefore, the conversion unit 145 performs conversion processing.

[0128] First, the selected foreground image candidate is called image A. Here, the camera position and posture are represented by the external parameter matrix ((R|T)). The camera posture corresponding to image A is camera posture R A That is, the camera posture R A This corresponds to the “R” of the camera position and posture “(R|T)” corresponding to image A. The object setting point corresponding to image A is the object setting point p A The object setting plane corresponding to the image A is the object setting plane P A In addition, the overlapping position p corresponding to the background image B and plane P B The camera pose R corresponding to the background image has been obtained. B is an unknown value.

[0129] (Step S43) The conversion unit 145 calculates the basic vectors of the object arrangement position coordinate system C in the image A. Here, the object arrangement position coordinate system C is exemplified.

[0130] Fig. 22 1 is a diagram showing an example of an object setting position coordinate system. In the object setting position coordinate system C, the vector connecting the camera and the object setting point is the Z axis. The conversion unit 145 performs projection conversion so that the camera posture R in the object setting position coordinate system C is A The camera pose R in the object position coordinate system C B equal.

[0131] The calculation of the basic vectors of the coordinate system C for setting the position of the object in the image A is described in detail. Here, the world coordinates of the camera position are O A. Object setting point p A The world coordinate is X A By O A The normal vector of the object sets the plane P A The world coordinate of the intersection point is Y A .

[0132] The X-axis direction of the basic vector of the object setting position coordinate system C is expressed by equation (3).

[0133] [Formula 3]

[0134]

[0135] The Y-axis direction of the basic vector of the object setting position coordinate system C is expressed by equation (4).

[0136] [Formula 4]

[0137]

[0138] The Z-axis direction of the basic vector of the object setting position coordinate system C is expressed by equation (5).

[0139] [Formula 5]

[0140]

[0141] (Step S44) The conversion unit 145 calculates the basic vector of the object setting position coordinate system C in the background image. Here, the camera position in the camera coordinate system is O B . Overlap position p B The world coordinate is X B By O B The normal vector of the plane P B The world coordinate of the intersection point is Y B .

[0142] The X-axis direction of the basic vector of the object setting position coordinate system C is expressed by equation (6).

[0143] [Formula 6]

[0144]

[0145] The Y-axis direction of the basic vector of the object setting position coordinate system C is expressed by equation (7).

[0146] [Formula 7]

[0147]

[0148] The Z-axis direction of the basic vector of the object setting position coordinate system C is expressed by equation (8).

[0149] [Formula 8]

[0150]

[0151] The matrix obtained by vertically concatenating the basic vectors in image A is T A . T A It is expressed by formula (9).

[0152] [Formula 9]

[0153]

[0154] The matrix obtained by vertically concatenating the basic vectors in the background image is T B . T B It is expressed by formula (10).

[0155] [Formula 10]

[0156]

[0157] The camera pose R of image A in the object setting position coordinate system C A1 It is expressed by formula (11).

[0158] [Formula 11]

[0159]

[0160] The camera pose R of the background image in the object setting position coordinate system C B1 It is expressed by formula (12).

[0161] [Formula 12]

[0162]

[0163] The camera pose R of the background image in the world coordinate system B It is expressed by formula (13). In addition, the camera posture R B is the rotation matrix.

[0164] [Formula 13]

[0165] R B =T A R B1 …(13) The transformation matrix T of the camera posture AB It is expressed by formula (14).

[0166] [Formula 14]

[0167]

[0168] (Step S45) The conversion unit 145 uses equation (15) to calculate the homography matrix HAB . In addition, K is the camera's intrinsic parameter matrix.

[0169] [Formula 15]

[0170] H AB =KT AB K …(15)

[0171] (Step S46) The conversion unit 145 performs a projective conversion on the image A using equation (16). The projective conversion in equation (16) is also called a homography conversion. In addition, x and y are pixel coordinates of the image A. new ,y new are the pixel coordinates of image A after projection transformation.

[0172] [Formula 16]

[0173]

[0174] In this way, the conversion unit 145 performs projection conversion on the image A in the shooting direction in the background image. In other words, the conversion unit 145 performs projection conversion on the image A in the shooting direction of the camera that generated the background image.

[0175] In addition, when the shooting direction of the background image is the same as the shooting direction of the image A, the image A is not projectively converted even if steps S43 to S46 are performed.

[0176] Fig.23 (A) and (B) are diagrams (part 1) for explaining projection transformation. Fig.23 (A) shows image 60 as image A. Fig.23 (B) shows an image 61 as a background image.

[0177] The angle θ corresponding to the image 60 and the angle θ corresponding to the image 61 B However, the shooting direction corresponding to image 60 is 90 degrees different from the shooting direction corresponding to image 61. Therefore, the visual conditions are different. Therefore, the conversion unit 145 performs projection conversion on image A.

[0178] Fig.24 This is a diagram for explaining projection transformation (part 2). Fig.24 The image A after the projective conversion is shown, that is, the image 62. By performing the projective conversion in this way, the visual state of the foreground image becomes the same as the visual state of the background image.

[0179] After step S46, the process proceeds to step S51.

[0180] (Step S51) The conversion unit 145 determines whether the distance between the camera and the object setting plane is different in the image A and the background image. Here, for example, the original image as the foreground image of the image A is an image generated by an RGB-D camera. Therefore, the distance corresponds to each pixel of the image A (i.e., the foreground image). In addition, the point cloud data corresponding to each pixel of the background image corresponds to the background image. Therefore, the distance is determined by the Z coordinate shown in the point cloud data.

[0181] When the distance is different between image A and the background image, it means that the ratio is different between image A and the background image. When the distance is different between image A and the background image, the process proceeds to step S52. When the distance is the same between image A and the background image, the process proceeds to step S53.

[0182] (Step S52 ) The conversion unit 145 performs scale conversion on the image A.

[0183] The scale conversion is described in detail. First, let the distance in image A (i.e., the vector representing the shooting direction and the object setting plane P) be A The distance between the intersection point and the camera position is d A The distance in the background image (i.e., the vector representing the shooting direction and the plane P B The distance between the intersection point and the camera position is d B In addition, in the following description, image A is projectively converted.

[0184] The conversion unit 145 uses affine transformation to perform scale conversion on the image A. Specifically, the conversion unit 145 performs scale conversion on the image A using equation (17).

[0185] [Formula 17]

[0186]

[0187] Here, image A after scale conversion is illustrated.

[0188] Fig.25 2 is a diagram showing an example of an image after scale conversion. The conversion unit 145 performs scale conversion on the image 62 as the image A. As a result, the object 20 included in the image 62 becomes smaller.

[0189] In the above, the case where the image A is projectively converted is described. When the shooting direction in the image A is the same as the shooting direction in the background image, the conversion unit 145 does not perform the projective conversion on the image A. In addition, in the above, the case where the image A after the projective conversion is scaled is described. The conversion unit 145 may also perform the scale conversion on the image A which has not been projectively converted.

[0190] (Step S53) The selection unit 144 determines whether all the foreground image candidates have been selected. If all the foreground image candidates have been selected, the process proceeds to step S54. If all the foreground image candidates have not been selected, the process proceeds to step S42.

[0191] (Step S54) The selection unit 144 selects the best foreground image candidate from the one or more foreground image candidates as the foreground image to be superimposed. In other words, the selection unit 144 selects the image with the smallest change in viewpoint due to image conversion from the one or more foreground image candidates.

[0192] Here, an index d(P, Q) is introduced to indicate the similarity between the rotation matrix P and the rotation matrix Q. The smaller the value of the index d(P, Q), the more similar the rotation matrix P and the rotation matrix Q are. In addition, a small value of the index d(P, Q) indicates that even if the camera posture changes, the change in the viewpoint is small. The index d(P, Q) can be expressed using equation (18).

[0193] [Formula 18]

[0194]

[0195] x∈Point on the unit sphere

[0196] Regarding the indicator d(P, Q), points are randomly sampled on the unit sphere, and the distance between the coordinates of each point after the transformation based on the rotation matrix P and the coordinates of each point after the transformation based on the rotation matrix Q is calculated, thereby simulating the calculation of the indicator d(P, Q).

[0197] In the process of selecting the optimal foreground image candidate, “P” in the index d(P, Q) is replaced by the camera pose R A In addition, the “Q” of the index d(P, Q) is replaced by the camera posture R B .

[0198] When there are N foreground image candidates, the selection unit 144 selects image i which is the optimal foreground image candidate using equation (19).

[0199] [Formula 19]

[0200]

[0201] (Step S55) The superimposing unit 146 superimposes the selected image i (ie, the foreground image) on the background image. Specifically, the superimposing unit 146 superimposes the foreground image on the background image at a superimposing position p. B .

[0202] Here, in the above description, for ease of understanding, the case where the projection conversion and the scale conversion are performed on the image A is described. However, in reality, the foreground image showing the object 20 included in the image A is projectedly converted and the scale conversion is performed.

[0203] The process from projection conversion to overlay is briefly illustrated.

[0204] Fig.26 It is a diagram showing the flow from projection conversion to superposition. Fig.26 The foreground image 70 is shown. The data generating unit 140 performs a projection conversion on the foreground image 70. The data generating unit 140 performs a scale conversion on the foreground image 70. In other words, the data generating unit 140 converts the scale of the foreground image 70 into the scale of the background image 71. The data generating unit 140 overlaps the foreground image 70 on the background image 71. Thus, a composite image is generated. The composite image is used as learning data. In addition, the information registered in the measurement table 111 can also be matched with the composite image. By matching the information with the composite image, the learning data can be filtered. In addition, by matching the information with the composite image, in the case of an imbalance in the conditions, the learning data that corrects the imbalance in the conditions can be extracted. In addition, the correct answer information can also be matched with the composite image.

[0205] The information processing device 100 associates the angle θ corresponding to the background image with B The foreground image of the same or similar angle θ is superimposed on the background image. Therefore, the synthesized image obtained by superimposing the foreground image on the background image becomes real data. Therefore, the information processing device 100 can generate real data.

[0206] In addition, when the shooting direction of the foreground image is different from that of the background image, the information processing device 100 performs a projection conversion on the foreground image. As a result, the direction of the foreground image becomes the same as the shooting direction of the background image. Then, the information processing device 100 overlaps the foreground image after the projection conversion on the background image. Therefore, the composite image obtained by overlapping the foreground image with the background image becomes more realistic data. Therefore, the information processing device 100 can generate more realistic data.

[0207] In addition, when the ratio of the foreground image is different from that of the background image, the information processing device 100 performs a ratio conversion on the foreground image. Thus, the foreground image is converted to the same ratio as the background image. Then, the information processing device 100 overlaps the foreground image after the ratio conversion on the background image. Therefore, the composite image obtained by overlapping the foreground image with the background image becomes more realistic data. Therefore, the information processing device 100 can generate more realistic data.

[0208] The information processing device 100 selects the best foreground image candidate from a plurality of foreground image candidates as the foreground image to be overlapped. Then, the information processing device 100 overlaps the foreground image as the selected foreground image candidate with the background image. Therefore, the synthesized image obtained by overlapping the foreground image with the background image becomes more optimal data. Therefore, the information processing device 100 can generate more optimal data.

[0209] Description of Reference Numerals

[0210] 10 robot arm, 11 camera, 20 object, 21 object setting point, 30 belt conveyor, 31 object, 32 camera, 40 image, 41 rectangular area, 42 mask image, 43 foreground image, 50 image, 51 image, 60 image, 61 image, 62 image, 70 foreground image, 71 background image, 100 information processing device, 101 processor, 102 volatile storage device, 103 non-volatile storage device, 104 input device, 105 display device, 110 storage unit, 111 measurement table, 120 acquisition unit, 130 foreground image generation unit, 131 area calculation unit, 132 mask image generation unit, 133 foreground image generation unit, 140 data generation unit, 141 overlapping position determination unit, 142 calculation unit, 143 search unit, 144 selection unit, 145 conversion unit, 146 overlapping unit.

Claims

1. An information processing device, wherein: The information processing device comprises: an acquisition unit that acquires a foreground image corresponding to an angle of the foreground image and representing an object, a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position when generating the background image; an overlapping position determining unit, which determines an overlapping position from the background image; a calculation unit that uses the point cloud data to calculate a plane that is a region including the overlapping position, and calculates a background image angle that is an angle between a straight line connecting the camera position and the overlapping position and the calculated plane; as well as an overlapping portion, which overlaps the foreground image with the background image, The foreground image angle is the angle between the plane on which the object is set when the original image of the foreground image is generated and the straight line indicating the direction in which the object is photographed, that is, the photographing direction, and is the same as or similar to the background image angle.

2. The information processing device according to claim 1, wherein: The information processing device further includes a conversion unit that performs projective conversion on the foreground image in a shooting direction in the background image.

3. The information processing device according to claim 1 or 2, wherein: The information processing device further includes a conversion unit that converts the ratio of the foreground image into the ratio of the background image.

4. The information processing device according to any one of claims 1 to 3, wherein: The information processing device further includes a selection unit. The acquisition unit acquires a plurality of foreground image candidates associated with angles that are the same as or similar to the angle of the background image. The selection unit selects an optimal foreground image candidate from among the plurality of foreground image candidates as the foreground image to be superimposed, The foreground image to be superimposed on the background image is a selected foreground image candidate.

5. A generation method, wherein: The information processing device performs the following processing: obtaining a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position when generating the background image, determining an overlapping position from said background image, Using the point cloud data, calculating a plane as a region including the overlapping position, calculating a background image angle, the background image angle being the angle between a straight line connecting the camera position and the overlapping position and the calculated plane, Acquire a foreground image corresponding to the angle of the foreground image and representing the object, wherein the angle of the foreground image is the same as or similar to the angle of the background image, Overlaying the foreground image on the background image, The foreground image angle is an angle between a plane on which the object is placed when an original image of the foreground image is generated and a straight line indicating a direction in which the object is photographed, that is, a photographing direction.

6. A generating program, wherein: The generation program causes the information processing device to execute the following processing: obtaining a background image, point cloud data representing three-dimensional coordinates corresponding to each pixel of the background image, and a camera position when generating the background image, determining an overlapping position from said background image, Using the point cloud data, calculating a plane as a region including the overlapping position, calculating a background image angle, the background image angle being the angle between a straight line connecting the camera position and the overlapping position and the calculated plane, Acquire a foreground image corresponding to the angle of the foreground image and representing the object, wherein the angle of the foreground image is the same as or similar to the angle of the background image, Overlaying the foreground image on the background image, The foreground image angle is an angle between a plane on which the object is placed when an original image of the foreground image is generated and a straight line indicating a direction in which the object is photographed, that is, a photographing direction.

Images