Information processing device, object identification device and object identification system

Abstract

The present invention discloses an information processing device, an object recognition device and an object recognition system, which can generate learning data used for a recognition model, and the information processing device has a memory and a processor. The memory stores a background image that does not contain a detection object, a detection object image that is an image of the detection object, and label information of the detection object representing the detection object image. The processor performs trapezoidal correction on the background image, overlaps the detection object image with the background image after trapezoidal correction to generate the composite image, and generates learning data based on the composite image and the label information.

Description

[0001] This application claims priority from Japanese patent application No. JP2019-096182, filed on May 22, 2019, and cites the contents of the aforementioned application, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0002] The embodiments of the present invention relate to an information processing device, an object recognition device and an object recognition system. Background Art

[0003] An apparatus has been put into practical use that recognizes (labels) detection objects from images (recognition object images) captured of detection objects (commodities, articles, etc.) placed on a container having a shelf based on a recognition model created in advance.

[0004] The recognition model is created in advance through machine learning based on learning data. Machine learning is, for example, a technique such as a neural network for segmentation. Learning data is data including questions and answers. For example, to create a recognition model for recognizing a detection object from an image, the following learning data is used: an image containing a detection object is used as a question, and the area of ​​the detection object in the image and the information representing the detection object (label information) are used as answers. The learning data is generated, for example, by manually adding the area where the detection object exists and the label information to the image containing the detection object. In this way, the generation of learning data takes time, so there is a method of generating learning data by overlapping an image of the detection object (detection object image) with a background image that does not contain the detection object.

[0005] The detection object image is an image of a specified face (face, appearance surface) of the detection object. The appearance surface of the detection object is any face in the appearance that can be seen when the detection object is arranged in a container. For example, the detection object image is an image captured by a camera when the camera is facing the appearance surface of the detection object. The background image is, for example, an image of a container and its vicinity where the detection object is not arranged. When capturing the background image, the angle of the optical axis of the optical system of the camera relative to the container may not be the angle that the appearance surface of the detection object is facing when the detection object is arranged in the container. There is a problem that the detection object image cannot be simply overlapped with the background image captured in this state. Summary of the invention

[0006] In view of the above problems, the technical problem to be solved by the present invention is to provide an information processing device, an object recognition device and an object recognition system that can generate learning data used for a recognition model.

[0007] To solve the above problem, an embodiment of the present invention provides an information processing device comprising: a memory and a processor. The memory stores a background image that does not include a detection object, a detection object image that is an image of the detection object, and label information of the detection object representing the detection object image. The processor performs trapezoidal correction on the background image, overlaps the detection object image with the background image after trapezoidal correction to generate the composite image, and generates learning data based on the composite image and the label information.

[0008] According to such a configuration, it is possible to generate learning data used for a recognition model.

[0009] For the information processing device, in a possible implementation, the processor transforms the projection of the background image into the same projection mode as that of the detection object image, and performs trapezoidal correction on the background image after the projection transformation.

[0010] According to such a configuration, it is possible to easily collect learning data for machine learning.

[0011] For the information processing device, in a possible implementation, the processor corrects the detection object image based on a preset synthesis parameter, and overlaps the corrected detection object image with the background image to generate the synthesized image.

[0012] According to such a configuration, it is possible to easily collect learning data for machine learning.

[0013] In a possible implementation of the information processing device, the processor generates the learning data in a manner of associating a region of the detection object in the synthetic image with label information corresponding to the detection object.

[0014] According to such a configuration, a high-precision recognition model can be easily created.

[0015] Another embodiment of the present invention provides an object identification device, comprising: a memory, storing a background image that does not include a detection object, a detection object image that is an image of the detection object, and label information of the detection object representing the detection object image; and a processor, wherein the processor performs trapezoidal correction on the background image, overlaps the detection object image with the background image after trapezoidal correction to generate a composite image, generates learning data based on the composite image and the label information, creates a recognition model based on the generated learning data, and identifies the detection object in the recognition object image based on the recognition object image mapped with the detection object and the recognition model.

[0016] According to such a configuration, the detection target can be detected from the recognition target image with high accuracy.

[0017] For the object recognition device, in a possible implementation, the processor transforms the background image into a projection mode that is the same as that of the detection object image, and performs a trapezoidal correction on the background image after the projection transformation.

[0018] According to such a configuration, it is possible to easily collect learning data for machine learning.

[0019] For the object identification device, in one possible implementation, the processor transforms the projection of the identification object image into the same projection mode as the detection object image, performs trapezoidal correction on the identification object image after the projection transformation, and identifies the detection object in the identification object image based on the identification object image after the trapezoidal correction and the recognition model.

[0020] According to such a configuration, the detection target can be detected from the recognition target image with high accuracy.

[0021] For the object identification device, in a possible implementation, the processor corrects the detection object image based on a preset synthesis parameter, and overlaps the corrected detection object image with the background image to generate the synthesized image.

[0022] According to such a configuration, it is possible to easily collect learning data for machine learning.

[0023] Regarding the object recognition device, in a possible implementation, the processor generates the learning data in a manner of associating a region of the detection object in the synthetic image with label information corresponding to the detection object.

[0024] According to such a configuration, a high-precision recognition model can be easily created.

[0025] A third embodiment of the present invention provides an object recognition system, comprising a photographing device and an object recognition device, wherein the photographing device photographs an recognition object image mapped with a detection object and provides the recognition object image to the object recognition device, wherein the object recognition device comprises: a memory, storing a background image that does not include a detection object, a detection object image that is an image of the detection object, and label information of the detection object representing the detection object image; and a processor, wherein the processor performs trapezoidal correction on the background image, overlaps the detection object image with the background image after trapezoidal correction to generate a composite image, generates learning data based on the composite image and the label information, creates a recognition model based on the generated learning data, and recognizes the detection object in the recognition object image based on the recognition object image mapped with the detection object and the recognition model.

[0026] According to such a configuration, the detection target can be detected from the recognition target image with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Below, the information processing device, the object identification device and the object identification system involved in the embodiment are described with reference to the accompanying drawings. When considered in conjunction with the accompanying drawings, by referring to the detailed description below, the present invention can be more completely and better understood and many of the accompanying advantages can be easily known, but the drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention, wherein:

[0028] Figure 1 is a diagram schematically showing a configuration example of an article identification system according to an embodiment;

[0029] Figure 2 is a diagram for explaining a configuration example of an information processing device according to an embodiment;

[0030] Figure 3 is an explanatory diagram for explaining an example of the state of a photographing device when a background image is acquired according to an embodiment;

[0031] Figure 4 is an explanatory diagram for explaining an example of a background image according to an embodiment;

[0032] Figure 5 is an explanatory diagram for explaining an example of an identification target image according to an embodiment;

[0033] Figure 6 is an explanatory diagram for explaining an example of the state of a photographing device when acquiring a detection target image according to an embodiment;

[0034] Figure 7 is an explanatory diagram for explaining an example of a detection target image involved in one embodiment;

[0035] Figure 8 is a diagram for explaining an example of an operation in a learning phase of an information processing device according to an embodiment;

[0036] Fig. 9 is an explanatory diagram for explaining an example of projective transformation of a background image according to an embodiment;

[0037] Fig.10 is an explanatory diagram for explaining an example of trapezoidal correction of a background image according to an embodiment;

[0038] Fig.11is an explanatory diagram for explaining an example of a composite image according to an embodiment;

[0039] Fig.12 is a diagram for explaining an example of operations in the inference phase of an information processing device according to an embodiment;

[0040] Fig.13 is an explanatory diagram for explaining an example of projective transformation and trapezoidal correction of an identification target image according to an embodiment; and

[0041] Fig.14 This is an explanatory diagram for explaining an example of a recognition result obtained by an information processing device according to an embodiment.

[0042] Description of Reference Numerals

[0043] 1...item identification system, 2...container, 3...goods, 11...back panel, 12...shelf, 21...goods shelf photography device, 22...network, 23...information processing device, 31...system controller, 32...display unit, 33...operation interface, 34...communication interface, 41...processor, 42...memory. DETAILED DESCRIPTION

[0044] Hereinafter, embodiments will be described with reference to the accompanying drawings.

[0045] Figure 1 This is an explanatory diagram for explaining an example of a schematic configuration of an article identification system 1 according to an embodiment.

[0046] First, the outline of the article recognition system 1 will be described.

[0047] The article recognition system 1 is a system that captures images of display cabinets 2 set up in a predetermined location such as a store or warehouse and commodities (articles) 3 as detection targets placed on the cabinets 2, and performs labeling processing to recognize the detection targets from the captured images.

[0048] The container 2 includes a back panel 11 and a shelf 12 protruding from the back panel 11 .

[0049] The back plate 11 is a support member that supports the shelf 12. The back plate 11 is configured in a plate shape that stands vertically, for example.

[0050] The shelf 12 is a shelf plate that supports the commodities 3. The shelf 12 is configured to protrude in the horizontal direction from the back plate 11, for example.

[0051] The article recognition system 1 includes a commodity shelf imaging device 21 and an information processing device 23 connected via a network 22 .

[0052] The commodity shelf photographing device 21 is a camera having an optical system and a photographing element. The commodity shelf photographing device 21 converts the light formed by the optical system into an image (image data) by using the photographing element. The commodity shelf photographing device 21 provides the acquired image to the information processing device 23. The commodity shelf photographing device 21 is configured as a surveillance camera installed in a store, warehouse, etc. where the container 2 is configured. In addition, the commodity shelf photographing device 21 can be a camera for photographing commodities installed near the container 2. In addition, the commodity shelf photographing device 21 can be configured as a device having a moving mechanism that moves in a store or warehouse and a camera installed in the moving mechanism.

[0053] The information processing device 23 performs object detection using machine learning. Figure 2 2 is an explanatory diagram for explaining an example of the configuration of the information processing device 23. The information processing device 23 includes a system controller 31, a display unit 32, an operation interface 33, and a communication interface 34.

[0054] The system controller 31 controls the information processing device 23. The system controller 31 includes a processor 41 and a memory 42, for example.

[0055] The processor 41 is a computing element that performs arithmetic processing. The processor 41 is, for example, a CPU. In addition, the processor 41 may be, for example, a configuration including a CPU and a GPU. The processor 41 performs various processes based on data such as programs stored in the memory 42. The processor 41 functions as a control unit that can perform various actions by executing programs stored in the memory 42.

[0056] The memory 42 is a storage medium that stores programs and data used in the programs. In addition, the memory 42 also functions as a working memory. That is, the memory 42 temporarily stores data being processed by the processor 41 and programs executed by the processor 41.

[0057] The display unit 32 includes a display, and displays a screen based on an image signal input from a display control unit such as the system controller 31 or a graphic controller (not shown). For example, a screen for various settings of the information processing device 23 (such as a setting screen for synthesis parameters described later) is displayed on the display of the display unit 32.

[0058] The operation interface 33 is connected to an operation member (not shown). The operation interface 33 provides an operation signal corresponding to the operation of the operation member to the system controller 31. The operation member is, for example, a touch sensor, a numeric keypad, a power key, a paper feed key, various function keys or a keyboard. The touch sensor obtains information indicating a position specified in a certain area. The touch sensor is configured as a touch panel integrated with the display unit 32, and inputs a signal indicating a touched position on the screen displayed by the display unit 32 to the system controller 31.

[0059] The communication interface 34 is an interface for communicating with other devices. The communication interface 34 is used, for example, for communicating with the commodity shelf photography device 21 and an upper (superior) device (external device). The communication interface 34 wirelessly communicates with other devices in accordance with standards such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). In addition, the communication interface 34 can be configured as a LAN connector, etc.

[0060] Next, the image acquired by the commodity shelf imaging device 21 will be described.

[0061] The commodity shelf shooting device 21 is usually required to shoot a wide range. For this reason, the optical system of the commodity shelf shooting device 21 uses an optical system with a very large diagonal field of view (extremely short focal length). For example, the optical system of the commodity shelf shooting device 21 uses an optical system that adopts a first projection method (for example, an equidistant projection method, a stereoscopic projection method, an equi-stereoscopic projection method, or an orthographic projection method, etc., so-called fisheye projection methods) and a short focal length. It should be noted that in the following description, the optical system of the commodity shelf shooting device 21 is taken as an example of an optical system of an equidistant projection method.

[0062] Figure 3 2 is an explanatory diagram for explaining the state of the commodity shelf photographing device 21 when photographing a background image. Figure 3 As shown, the commodity shelf photographing device 21 obtains a background image by photographing the container 2 and its vicinity when the commodities 3 are not arranged in the container 2. Figure 3 As shown, the angle of the optical system of the commodity shelf photographing device 21 relative to the optical axis of the container 2 when photographing the background image is an angle directly facing the upper side of the commodity 3 (the side opposite to the side contacting the shelf 12) when the commodity 3 to be detected is not arranged in the container 2. That is, the optical system of the commodity shelf photographing device 21 forms an angle with the side of the shelf 12 of the container 2 where the commodities are arranged.

[0063] Figure 4 Indicated in Figure 3 The example of the background image captured by the commodity shelf imaging device 21 in the state shown is an image captured by the imaging device having an optical system of the first projection method (equidistant projection method in this example). That is, the background image is an image of the equidistant projection method.

[0064] It should be noted that the imaging device for acquiring the background image may be a different imaging device from the commodity shelf imaging device 21. In addition, the background image may be an image captured by an imaging device having an optical system of a projection method such as stereoscopic projection, equi-stereoscopic projection or orthographic projection.

[0065] Furthermore, the commodity shelf imaging device 21 acquires a recognition target image by imaging the container 2 and its vicinity in a state where the commodities 3 are arranged in the container 2 .

[0066] Figure 5 Indicated in Figure 3 An example of an image to be identified captured by the commodity shelf imaging device 21 when the commodity 3 is arranged in the container 2 in a state. The image to be identified is an image captured by an imaging device having an optical system of a first projection method (in this example, an equidistant projection method). That is, the image to be identified is an image of an equidistant projection method.

[0067] Next, images and learning data used in the information processing device 23 will be described.

[0068] The information processing device 23 includes an inference phase for inferring an object in an image based on a recognition model created in advance, and a learning phase for creating and updating a recognition model based on learning data (teaching data).

[0069] In the inference stage, the information processing device 23 performs labeling processing to identify the commodity 3 as the detection target from the recognition target image obtained by the commodity shelf imaging device 21 based on the recognition model created in advance. That is, the information processing device 23 performs labeling processing based on the recognition model to detect "where" and "what" in the recognition target image.

[0070] Furthermore, in the learning phase, the information processing device 23 performs a machine learning process of creating and updating a recognition model through machine learning based on the learning data.

[0071] Learning data is data that contains questions and answers. For example, when creating a recognition model for recognizing a detection object from an image, the following learning data is used: an image containing the detection object is used as the question, and the position of the detection object in the image and information representing the detection object (label information) are used as the answer.

[0072] The labeling process is described in, for example, "SSD: Single Shot MultiBox Detector (see https: / / arxiv.org / abs / 1512.02325)". In this method, a recognition model is created by machine learning based on learning data including an image, bounding box information indicating a position in the image, and label information selected from a finite subset.

[0073] The information processing device 23 generates learning data by superimposing an image of a detection target (detection target image) on a background image acquired in advance.

[0074] The detection object image is an image of a predetermined surface of the commodity to be detected. The predetermined surface is called an appearance surface. An appearance surface is any surface of the appearance that can be seen when the commodity is arranged in the container 2. In this example, it is assumed that the top surface of the commodity 3 arranged on the shelf 12 of the container 2 (the opposite side of the side contacting the shelf 12) is the appearance surface for description.

[0075] Figure 6 is an explanatory diagram for explaining the state of the camera when capturing the image of the detection object. The camera is facing the appearance surface of the commodity 3, and the camera uses the optical system with a second projection method (for example, a central projection method) to capture the commodity 3, thereby obtaining the detection object image. More specifically, Figure 6 As shown, the inspection target image is an image captured by the imaging device in a state where the optical axis of the optical system of the imaging device is perpendicular to the outer surface of the commodity 3 .

[0076] Figure 7 Indicated in Figure 6 An example of an image of the detection object captured by a camera in the state shown. The detection object image is a rectangular image captured by a camera having an optical system with a second projection mode (in this example, a center projection mode). That is, the background image is an image of the center projection mode. In addition, the detection object image is an image after the area of ​​the appearance surface of the commodity 3 is trimmed and the area outside the appearance surface is removed. In addition, label information is attached to the detection object image. The label information is attached to the detection object image when the detection object image is captured. In addition, the label information can be attached together or individually to the detection object image that is captured and saved.

[0077] The imaging device that acquires the detection target image is, for example, an imaging device different from the commodity shelf imaging device 21. It should be noted that the imaging device that acquires the detection target image may be the commodity shelf imaging device 21.

[0078] Next, the operation of the information processing device 23 in the learning phase will be described.

[0079] As described above, in the learning phase, the information processing device 23 performs a machine learning process to create and update a recognition model by machine learning based on the learning data. To this end, first, the information processing device 23 generates learning data.

[0080] The background image captured under the above conditions cannot be simply overlapped with the detection object image. This is because of the difference in image height caused by the difference in projection method between the background image and the detection object image and the difference in distance between the optical axis and the shooting surface. For this reason, the information processing device 23 transforms the background image through image processing, and overlaps the detection object image on the transformed background image, thereby generating learning data.

[0081] The processor 41 of the system controller 31 of the information processing device 23 acquires a plurality of background images in advance and stores them in the memory 42. In addition, the processor 41 acquires a plurality of detection target images to which label information is added in advance and stores them in the memory 42. In addition, the processor 41 acquires parameters (synthesis parameters) used for image processing of the detection target images in advance and stores them in the memory 42.

[0082] Figure 8 It is an explanatory diagram for explaining the machine learning processing in the information processing device 23.

[0083] First, the processor 41 reads a background image from the memory 42 (ACT11). The processor 41 reads a detection object image from the memory 42 (ACT12). The processor 41 can be configured to randomly read a background image and a detection object image from the memory 42, or can be configured to read a background image and a detection object image selected by the operation interface 33 from the memory 42. In addition, the processor 41 can also be configured to read a background image and a plurality of detection object images from the memory 42.

[0084] The processor 41 performs projection transformation to transform the projection mode of the read background image (ACT 13). For example, the processor 41 transforms the background image of the first projection mode (equidistant projection mode) into an image of the second projection mode (central projection mode) of the projection mode of the detection object image. Figure 4 The background image shown is transformed into Fig. 9 The background image shown.

[0085] The processor 41 performs a trapezoidal correction on the background image transformed into the center projection mode (ACT 14). For example, the processor 41 detects a plurality of straight lines extending toward a vanishing point in the background image transformed into the center projection mode. The processor 41 converts the plurality of straight line detection results into Fig. 9 The background image shown is transformed into Fig.10 The background image shown. In addition, the processor 41 may be configured to perform a keystone correction on the background image based on preset parameters.

[0086] The processor 41 reads the synthesis parameters from the memory 42 (ACT15). The synthesis parameters are parameters used in image processing (enlargement and reduction, rotation, hue correction, brightness correction, contrast correction, blurring, etc.) performed on the detection object image. The processor 41 corrects the detection object image based on the synthesis parameters (ACT16). That is, the processor 41 performs image processing such as enlargement and reduction, rotation, hue correction, brightness correction, contrast correction, blurring, etc. on the detection object image read in ACT12 based on the synthesis parameters read from the memory 42.

[0087] Furthermore, the synthesis parameters may include parameters used in image processing (hue correction, brightness correction, contrast correction, blurring, etc.) performed on the background image. Fig.10 The background image is processed by hue correction, brightness correction, contrast correction and blur.

[0088] Next, the processor 41 overlaps the detection target image with the background image to generate a composite image (ACT 17). That is, the processor 41 overlaps the detection target image corrected in ACT 16 with the background image corrected in ACT 14 to generate a composite image. Fig.11 The composite image shown. The number of detection object images overlapped on the background image can be one or more. In addition, the number of detection object images overlapped on the background image can be a preset number or a random number. In addition, the position on the background image where the detection object image is overlapped can be a preset position or a random position. In addition, the processor 41 can overlap multiple detection object images on the background image.

[0089] Moreover, the synthesis parameter may be information indicating the position (synthesis position) at which the detection target image is overlapped on the background image, information indicating the number (synthesis number) of the detection target images overlapped on the background image, information indicating whether the detection target images are allowed to overlap with each other, etc. In this case, the processor 41 determines the synthesis position and synthesis number of the overlapping detection target images, and whether the overlap is allowed, based on the synthesis parameter.

[0090] Processor 41 generates learning data (ACT18). Processor 41 generates learning data based on the composite image, the position of the detection object image on the composite image, and the label information corresponding to the detection object image. That is, the question in the learning data is a composite image generated by overlapping the detection object image on the background image, and the answer in the learning data is the position on the background image where the detection object image is overlapped and the label information corresponding to the detection object image. Processor 41 generates learning data by using the position where the detection object image is overlapped on the composite image and the label information attached to the overlapped detection object image as the answer in the learning data. Processor 41 stores the generated learning data in memory 42 (ACT19).

[0091] Next, the processor 41 determines whether to update the recognition model (ACT20). When the processor 41 determines that the recognition model is not to be updated (NO in ACT20), the processor 41 moves to the processing of ACT11 and continues to collect learning data. In addition, when the processor 41 determines that the recognition model is to be updated (YES in ACT20), the processor 41 updates the recognition model based on the collected learning data (ACT21) and ends the processing. For example, when the recognition model is not stored in the memory 42 and an arbitrary amount of learning data is collected, the processor 41 creates a recognition model and stores it in the memory 42. In addition, each time an arbitrary amount of learning data is collected, the processor 41 updates the recognition model stored in the memory 42. In addition, the processor 41 can be configured to move to the processing of ACT11 after updating the recognition model stored in the memory 42 to continue collecting learning data.

[0092] As described above, the information processing device 23 includes a memory 42 and a processor 41, and the memory 42 stores a background image that does not include a detection object, a detection object image that is an image of the detection object, and label information of the detection object representing the detection object image. The processor 41 performs trapezoidal correction on the background image, and overlaps the detection object image on the background image after trapezoidal correction to generate a composite image. In addition, the processor 41 generates learning data based on the composite image and the label information corresponding to the detection object image overlapped on the background image. Thus, even if the angles are different during shooting, learning data can be generated based on the background image and the detection object image. Thus, it is easy to collect learning data used for machine learning. As a result, it is easy to create a high-precision recognition model.

[0093] In addition, the processor 41 transforms the background image projection into the same projection mode as the detection object image, and performs trapezoidal correction on the background image after the projection transformation for the generation of the composite image. Thus, even if the projection mode is different during shooting, learning data can be generated based on the background image and the detection object image. Thus, it is easy to collect learning data used for machine learning.

[0094] In addition, the processor 41 corrects the detection object image based on the preset synthesis parameters, and overlaps the corrected detection object image with the background image to generate a synthetic image. Thus, a synthetic image can be generated in which the differences in orientation, image size, hue, brightness, and contrast between the detection object image and the background image are corrected. Thus, learning data used for machine learning can be easily collected.

[0095] Next, the operation of the information processing device 23 in the inference phase will be described.

[0096] As described above, in the inference stage, the information processing device 23 performs labeling processing for recognizing the commodity 3 as the detection target from the recognition target image acquired by the commodity shelf imaging device 21 based on the recognition model created in advance.

[0097] Fig.12 This is an explanatory diagram for explaining labeling processing in the information processing device 23.

[0098] First, the processor 41 reads the recognition model from the memory 42 (ACT 31). In addition, the processor 41 starts the camera, that is, the commodity shelf imaging device 21 (ACT 32). The processor 41 determines whether the commodity shelf imaging device 21 performs imaging (ACT 33). If the processor 41 determines that the commodity shelf imaging device 21 performs imaging (YES in ACT 33), the processor 41 causes the commodity shelf imaging device 21 to perform imaging, and obtains, for example, Figure 5 The identification target image shown (ACT 34). When the processor 41 determines that the commodity shelf imaging device 21 is not to perform imaging (NO in ACT 33), the processor 41 proceeds to the processing of ACT 40 described later.

[0099] For example, the processor 41 determines that the commodity shelf photographing device 21 performs photographing at a preset time. In addition, the processor 41 may be configured to determine that the commodity shelf photographing device 21 performs photographing at a preset time interval. In addition, the processor 41 may be configured to determine that the commodity shelf photographing device 21 performs photographing based on the detection results of other sensors (such as motion sensors, etc.). In addition, the processor 41 may be configured to continuously perform photographing by the commodity shelf photographing device 21, and use a part of the continuously acquired images as the recognition target image.

[0100] The processor 41 performs projection transformation on the acquired recognition target image (ACT 35). The processor 41 transforms the acquired recognition target image of the first projection method (equidistant projection method) into an image of the second projection method (central projection method) which is the projection method of the image used to create the recognition model.

[0101] The processor 41 performs a trapezoidal correction on the recognition object image transformed into the central projection mode (ACT 36). For example, the processor 41 detects a plurality of straight lines extending toward a vanishing point in the recognition object image transformed into the central projection mode. The processor 41 performs a trapezoidal correction based on the plurality of straight line detection results. In addition, the processor 41 can be configured to perform a trapezoidal correction on the background image based on a preset parameter.

[0102] Through the processing of ACT35 and ACT36, the processor 41 Figure 5The identification object image of the equidistant projection method shown in the figure is transformed into the central projection method and transformed into the image after the keystone correction is implemented. Fig.13 The appearance of the commodity 3 to be detected is mapped to the same state as the detection object image. Fig.13 In the identified object image shown.

[0103] The processor 41 inputs the recognition target image after the projection transformation and trapezoidal correction into the recognition model (ACT 37). Thus, each area on the recognition target image where the detection target, i.e., the appearance surface of the commodity 3, is mapped is obtained. Fig.14 The recognition result (ACT38) is shown. Fig.14 In the recognition result, a label information is attached to each area of ​​the detected appearance surface of the commodity 3. In addition, the recognition result may be a recognition result in which information indicating the reliability of each label information is attached to each area of ​​the detected appearance surface of the commodity 3. That is, the reliability of each label information in a certain area A may be constituted as "label A = 0.8", "label B = 0.05", "label C = 0.02", and the reliability of each label information in area B may be constituted as "label A = 0.01", "label B = 0.05", "label C = 0.9".

[0104] The processor 41 outputs the obtained recognition result (ACT39) and determines whether to end the labeling process (ACT40). When the processor 41 determines that the labeling process is not to be ended (NO in ACT40), it moves to the process of ACT33, reacquires the recognition target image, and continues the labeling process. When the processor 41 determines that the labeling process is to be ended (YES in ACT40), the labeling process is ended.

[0105] As described above, the information processing device 23 that functions as an object identification device has a memory 42 and a processor 41, wherein the memory 42 stores a background image that does not include a detection object, a detection object image that is an image of the detection object, and label information of the detection object representing the detection object image. The processor 41 performs trapezoidal correction on the background image, and overlaps the detection object image on the background image after trapezoidal correction to generate a composite image. The processor 41 also generates learning data based on the label information corresponding to the composite image and the detection object image overlapped on the background image. The processor 41 then creates a recognition model based on the learning data. As a result, the information processing device 23 can easily create a high-precision recognition model.

[0106] In addition, the processor 41 recognizes the detection object in the recognition object image, that is, the commodity 3, based on the recognition object image to which the detection object is mapped and the recognition model. The detection object in the recognition object image, that is, the commodity 3, is recognized using the recognition model created in this way, so that the detection object can be detected from the recognition object image with high accuracy.

[0107] It should be noted that the above-mentioned information processing device 23 can be a structure that can be counted and has an increase or decrease in quantity within a specified time as a detection object. For example, the information processing device 23 can be applied to a system (shelf management system) for confirming the checkout and replenishment status of goods in a store. For example, the information processing device 23 can be applied to a system that detects fruits and vegetables, packaged goods (boxes or bags), beverages (cans, bottles or boxes), customers, store clerks, etc. based on images in a supermarket. In addition, for example, the information processing device 23 can be applied to a system that detects goods, customers, store clerks, etc. based on images in a specialty store. In addition, for example, the information processing device 23 can be applied to a system that detects food, tableware (plates, glassware, chopsticks), diners, store clerks, etc. based on images in a restaurant.

[0108] In addition, the information processing device 23 can be applied to a system for confirming the entry and exit of articles in a logistics warehouse (article inspection system). For example, the information processing device 23 can be applied to a system for detecting cardboard, containers (cages), vehicles (lifts or loaders), people, etc. based on images in a logistics warehouse.

[0109] In addition, the information processing device 23 can be applied to a picking system at a manufacturing site. For example, the information processing device 23 can be applied to a system that detects parts, tools, mechanical devices (robots) (assembly or picking), vehicles (lifts or loaders), people, etc. based on images at a manufacturing site.

[0110] Several embodiments of the present invention are described above, but the above embodiments are described as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made within the scope of the gist of the invention. These embodiments and their variations are all included in the scope and gist of the invention, and are included in the schemes recorded in the claims and their equivalents.

Claims

1. An information processing device, characterized in that: include: A memory storing a background image not including the detection object photographed by a photographing device at an angle not directly facing the surface of the detection object, a detection object image as an image of the detection object photographed by a photographing device in a state orthogonal to the surface of the detection object, and label information of the detection object representing the detection object image; as well as A processor, the processor performs trapezoidal correction on the background image, overlaps the detection object image on the background image after trapezoidal correction to generate a composite image, and generates learning data based on the composite image and the label information, wherein: The processor transforms the projection of the background image into the same projection mode as that of the detection object image, and performs trapezoidal correction on the background image after the projection transformation.

2. The information processing device according to claim 1, wherein: The processor corrects the detection object image based on a preset synthesis parameter, and overlaps the corrected detection object image with the background image to generate the synthesized image.

3. The information processing device according to claim 1 or 2, wherein: The processor generates the learning data in such a manner that a region of the detection object in the synthetic image is associated with label information corresponding to the detection object.

4. An object identification device, characterized in that: include: A memory storing a background image not including the detection object photographed by a photographing device at an angle not directly facing the surface of the detection object, a detection object image as an image of the detection object photographed by a photographing device in a state orthogonal to the surface of the detection object, and label information of the detection object representing the detection object image; as well as a processor, the processor performing trapezoidal correction on the background image, overlapping the detection object image on the trapezoidal-corrected background image to generate a composite image, generating learning data based on the composite image and the label information, creating a recognition model based on the generated learning data, and recognizing the detection object in the recognition object image based on the recognition object image mapped with the detection object and the recognition model, wherein The processor transforms the projection of the background image into the same projection mode as that of the detection object image, and performs trapezoidal correction on the background image after the projection transformation.

5. The object identification device according to claim 4, wherein: The processor transforms the recognition object image into a projection mode that is the same as that of the detection object image, performs trapezoidal correction on the recognition object image after the projection transformation, and identifies the detection object in the recognition object image based on the recognition object image after the trapezoidal correction and the recognition model.

6. The article identification device according to claim 4 or 5, wherein: The processor corrects the detection object image based on a preset synthesis parameter, and overlaps the corrected detection object image with the background image to generate the synthesized image.

7. The article identification device according to claim 4 or 5, wherein: The processor generates the learning data in such a manner that a region of the detection object in the synthetic image is associated with label information corresponding to the detection object.

8. The object identification device according to claim 6, wherein: The processor generates the learning data in such a manner that a region of the detection object in the synthetic image is associated with label information corresponding to the detection object.

9. An object recognition system, comprising a photographing device and an object recognition device, characterized in that: The photographing device photographs the recognition object image on which the detection object is mapped, and provides the recognition object image to the object recognition device, The object identification device comprises: a memory storing a background image not including the detection object photographed by a photographing device at an angle not directly facing the surface of the detection object, a detection object image as an image of the detection object photographed by the photographing device in a state perpendicular to the surface of the detection object, and label information of the detection object representing the detection object image; and A processor, wherein the processor performs trapezoidal correction on the background image, overlaps the detection object image on the background image after trapezoidal correction to generate a composite image, generates learning data based on the composite image and the label information, creates a recognition model based on the generated learning data, and recognizes the detection object in the recognition object image based on the recognition object image mapped with the detection object and the recognition model, wherein the processor performs a projection transformation of the background image into the same projection mode as that of the detection object image, and performs trapezoidal correction on the background image after the projection transformation.

Images