Learnt model generating method, user environment presuming method, learnt model generating apparatus, user environment presuming apparatus and learnt model generating system

Abstract

To provide a learnt model generating apparatus, a learnt model generating method, a user environment presuming apparatus, a user environment presuming program and a learnt model generating system that allow for improving the robustness of a learnt model to be used in recognition.SOLUTION: A learnt model generating method includes: acquiring a first model that learning processing has been done on an object of presuming with first image data indicative of an object of presuming in a first environment taken as learning data; acquiring second image data indicative of an object of presuming in a second environment for executing presumption; generating a second model based on the first model with the second image data taken as learning data; and outputting a learnt model based on the second model. The second image data includes an image assuming how to view the object of presuming in the second environment based on user-environment information about the second environment.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to Japanese Patent Application No. 2021-121958 (filed July 26, 2021), the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to a trained model generation method, a user environment estimation method, a trained model generation device, a user environment estimation device, and a trained model generation system. [Background technology]

[0003] Conventionally, a system is known that creates a trained model to be used when capturing an image of a part and performing image recognition on the part (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 155593 Summary of the Invention

[0005] A trained model generating method according to an embodiment of the present disclosure includes: acquiring a first model obtained by performing a learning process on an estimation target using first image data showing an estimation target in a first environment as training data; acquiring second image data showing the estimation target in a second environment in which estimation is performed; generating a second model based on the first model using the second image data as training data; and outputting a trained model based on the second model. The second image data includes an image that assumes how the estimation target will appear in the second environment based on user environment information about the second environment.

[0006] A user environment estimation method according to an embodiment of the present disclosure estimates a user environment, which is an environment in which data to be estimated is acquired, and includes outputting a result of estimating the user environment based on image data obtained by capturing an image of a predetermined object in the user environment as user environment information regarding the user environment.

[0007] A trained model generation device according to an embodiment of the present disclosure includes a control unit. The control unit acquires a first model that has undergone a learning process for an estimation target using first image data showing the estimation target in a first environment as training data. The control unit acquires second image data showing the estimation target in a second environment in which estimation is performed. The control unit generates a second model based on the first model using the second image data as training data. The control unit outputs a trained model based on the second model. The second image data includes an image that assumes how the estimation target will appear in the second environment based on user environment information about the second environment.

[0008] A user environment estimation device according to an embodiment of the present disclosure includes a control unit that estimates a user environment, which is an environment in which data to be estimated is acquired, and outputs a result of estimating the user environment based on image data obtained by capturing an image of a predetermined object in the user environment as user environment information regarding the user environment.

[0009] A trained model generation system according to an embodiment of the present disclosure includes a trained model generation device that executes the trained model generation method, and a user environment estimation device that executes the user environment estimation method. The trained model generation device acquires the user environment from the user environment estimation device. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example configuration of a trained model generation system according to an embodiment. [Diagram 2]FIG. 1 is a block diagram showing an example of the configuration of functional blocks of a trained model generation system according to an embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing a configuration example of a first environment as a standard environment. [Figure 4] FIG. 2 is a schematic diagram showing a configuration example of a second environment as a user environment. [Diagram 5] FIG. 13 is a schematic diagram showing an example configuration for photographing a marker in a second environment. [Figure 6] 1 is a flowchart illustrating an example of a procedure of a trained model generation method according to an embodiment. [Figure 7] 10 is a flowchart showing an example of a procedure for generating second image data based on user environment information. [Figure 8] FIG. 13 is a schematic diagram showing an example of a configuration in which a marker is illuminated with parallel light. [Figure 9A] This is an example image of a triangular pyramid marker illuminated with parallel light. [Figure 9B] This is an example image of a pyramidal marker illuminated with parallel light. [Figure 9C] This is an example image of a square prism marker illuminated with parallel light. [Figure 10] FIG. 13 is a schematic diagram showing an example of a configuration in which a marker is illuminated by a spotlight. [Figure 11] An example image of a pyramidal marker illuminated by a spotlight. [Figure 12] FIG. 13 is a schematic diagram showing an example configuration in which a marker is illuminated from two directions by spotlights. [Figure 13] This is an example image of a triangular pyramid marker illuminated by spotlights from two directions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] When an environment in which a trained model is used for recognition is different from an environment in which the trained model was created, the recognition accuracy may decrease. In other words, the robustness of the trained model used for recognition may decrease. According to the trained model generation method and the user environment estimation method according to an embodiment of the present disclosure, the robustness of the trained model used for recognition may be improved.

[0012] (Configuration of trained model generation system 1) As shown in FIG. 1, a trained model generation system 1 according to an embodiment includes a first trained model generation device 110 and a second trained model generation device 210. The trained model generation system 1 further includes a user environment estimation device 310, although this is not essential. The trained model generation system 1 further includes an imaging device 40, although this is not essential. The first trained model generation device 110 and the second trained model generation device 210 generate a trained model used to estimate an estimation target. The trained model is an inference algorithm configured to perform a specific calculation process on an input by applying built-in trained parameters and to be able to output a calculation result. The trained model is used to estimate an estimation target. The trained model is used, for example, to recognize a recognition target or to estimate a gripping position of a gripping target.

[0013] This trained model is set, preferred, or downloaded to a robot controller that controls a robot such as a collaborative robot, and is used when the robot recognizes a work object. Specifically, when the robot performs a task, the trained model can, for example, photograph an object in the work environment, and based on the photographed image, determine whether the photographed object is a work object such as a recognition object or a grasp object, or estimate the grasp position of the grasp object. Then, the robot can be controlled according to the determination result.

[0014] The first trained model generation device 110 generates a first trained model by learning using an image of a recognition target in a first environment or an image assuming how the recognition target appears in a standard environment as training data. In the present disclosure, the first environment is also referred to as a standard environment. That is, in the following description, the term "standard environment" can be replaced with the term "first environment". The first trained model generation device 110 may acquire an image of a recognition target in a standard environment from the imaging device 40. The standard environment may be an environment that reduces the influence on an image of a recognition target or an image assuming how the recognition target appears. In other words, the standard environment may be an environment with less noise than the user environment described later. In addition, in the standard environment, when a user uses a device such as a robot equipped with a trained model, factors that may vary for each location of use may be smaller than those in the user environment described later. In other words, the environmental information can be said to be an environment in which factors that may vary for each environment in which recognition is performed are small.

[0015] The second trained model generation device 210 acquires the first trained model from the first trained model generation device 110. The second trained model generation device 210 updates the first trained model by learning using as training data an image that assumes how a recognition target will appear in a second environment in which recognition is performed, to generate a second trained model. In the present disclosure, the second environment is also referred to as a user environment. That is, in the following description, the term "user environment" can be replaced with the term "second environment." The environment in which recognition is performed may be, for example, a place where a device such as a robot equipped with the final trained model is used.

[0016] The user environment is an environment different from the standard environment. In this embodiment, it is assumed that the appearance of the recognition target in the standard environment is a standard appearance. The appearance of the recognition target in the user environment is different from the standard appearance. It can be said that the difference from the standard appearance causes noise in the appearance. Therefore, it can be said that the difference between the user environment and the standard environment causes noise in the appearance of the recognition target.

[0017] The trained model generation system 1 according to the present embodiment can improve the recognition accuracy of the recognition target in each environment by learning based on the difference in the appearance of the recognition target in each environment. In other words, a model having high robustness against the difference in the environment can be generated. An example of the configuration of the trained model generation system 1 will be described below.

[0018] <First trained model generation device 110, second trained model generation device 210, and user environment estimation device 310> 1, the first trained model generation device 110 includes a first control unit 120 and a first storage unit 130. As shown in FIG 2, the first control unit 120 includes a standard environment object data generation unit 121 and a standard environment object recognition unit 122. The first storage unit 130 includes a first data holding unit 131.

[0019] 1, the second trained model generation device 210 includes a second control unit 220 and a second storage unit 230. As shown in FIG 2, the second control unit 220 includes a user environment object data generation unit 223 and a user environment object recognition unit 224. The second storage unit 230 includes a second data holding unit 232.

[0020] 1, the user environment estimation device 310 includes a third control unit 320 and a third storage unit 330. As shown in Fig. 2, the third control unit 320 includes a user environment acquisition unit 325 and a user environment estimation unit 326. The third storage unit 330 includes a third data storage unit 333 and a fourth data storage unit 334.

[0021] The first trained model generation device 110 and the second trained model generation device 210 may be configured as an integrated device. The user environment estimation device 310 may be configured as an integrated device with the first trained model generation device 110 or the second trained model generation device 210.

[0022] The standard environment object data generating unit 121 generates first image data including an image showing the recognition target in the standard environment. The standard environment object data generating unit 121 may obtain an image of the recognition target in the standard environment from the imaging device 40 as the image of the recognition target in the standard environment, and use the image as the first image data. The standard environment object data generating unit 121 may generate an image that assumes how the recognition target appears in the standard environment as the first image data. That is, the standard environment object data generating unit 121 may synthesize the first image data based on design data including CAD (Computer-Aided Design) data or drawings, taking into account the state of the standard environment. The standard environment object data generating unit 121 outputs the first image data to the standard environment object recognition unit 122. The standard environment object data generating unit 121 may store the first image data in the first data holding unit 131.

[0023] The standard environment object recognition unit 122 acquires the first image data from the standard environment object data generation unit 121. The standard environment object recognition unit 122 may acquire the first image data from the first data storage unit 131. The standard environment object recognition unit 122 executes learning of recognition in the standard environment using the first image data as teacher data, and generates a first trained model. The first trained model is also referred to as a first model. The standard environment object recognition unit 122 stores the first model generated by learning using the first image data as teacher data in the first data storage unit 131.

[0024] The user environment object data generating unit 223 generates second image data including an image showing the recognition target in the user environment. The user environment object data generating unit 223 may generate an image that assumes how the recognition target will look in the user environment as the second image data. The user environment object data generating unit 223 acquires information about the user environment generated by the user environment acquisition unit 325 and the user environment estimation unit 326 described later. The information about the user environment is also called user environment information. The user environment object data generating unit 223 generates the second image data based on the user environment information. That is, the user environment object data generating unit 223 may synthesize the second image data based on design data including CAD data or drawings, taking into account the state of the user environment. The user environment object data generating unit 223 outputs the second image data to the user environment object recognition unit 224. The user environment object data generating unit 223 may store the second image data in the second data holding unit 232. The user environment object data generating unit 223 may obtain an image of the recognition target in the user environment as the second image data.

[0025] The user environment object recognition unit 224 acquires the second image data from the user environment object data generation unit 223. The user environment object recognition unit 224 also acquires the first model from the first data storage unit 131. The user environment object recognition unit 224 executes learning using the second image data as teacher data and generates a second model based on the first model. In this embodiment, the user environment object recognition unit 224 updates the first model to generate the second model. The first model generated by the standard environment object recognition unit 122 and stored in the first data storage unit 131 is stored in the second data storage unit 232. The user environment object recognition unit 224 updates the first model by executing reading and writing to the first model stored in the second data storage unit 232, generates a second trained model, and stores it in the second data storage unit 232. The second trained model is also referred to as the second model. The user environment object recognition unit 224 outputs the second model as a trained model. In other words, the user environment object recognition unit 224 may output a trained model based on the second model. Furthermore, training may be performed on images captured in the user environment.

[0026] The first model may be stored in the first data storage unit 131. The user environment object recognition unit 224 may update the first model by executing reading and writing to the first model stored in the first data storage unit 131, generate a second model, and store it in the first data storage unit 131. The first data storage unit 131 and the second data storage unit 232 may be configured so as to be indistinguishable from each other, or may be configured as one unit.

[0027] In the above, an example of generating the second model by updating the first model is described, but the method of generating the second model is not limited to this. For example, the second model may be generated by connecting an additional trained model, which is different from the first model and has undergone a learning process regarding the user environment, to the first model. The additional trained model is also referred to as, for example, an adapter module.

[0028] The user environment acquisition unit 325 acquires information used to estimate the user environment. The information used to estimate the user environment is also referred to as user environment data. The user environment data may include an image captured in the user environment. Specifically, the user environment data may include, for example, an image of a recognition target captured in the user environment, an image of the surroundings of the recognition target captured in the user environment, or an image of a state in which the recognition target is not placed in the user environment. The user environment data may include known information such as lighting conditions in the user environment. The user environment acquisition unit 325 outputs the user environment data to the user environment estimation unit 326. The user environment acquisition unit 325 may store the user environment data in the third data holding unit 333.

[0029] The user environment estimation unit 326 estimates the user environment based on the user environment data. The user environment estimation unit 326 may acquire the user environment data from the user environment acquisition unit 325 or from the third data storage unit 333. The user environment may be specified by, for example, lighting conditions. The lighting conditions may include, for example, the position or number of lights, the type of light source, the brightness, luminance or illuminance of the lights, the color temperature of the lights, or the flicker of the lights. The type of the light source may be specified by whether the light source is parallel light or scattered light. The type of the light source may be specified as a point light source, a planar light source, or a ring light source. The user environment may be specified by, for example, the specifications or settings of the imaging device 40 used when the recognition is performed. The user environment may be specified by the conditions of objects present other than the recognition target, such as a stand on which the recognition target is placed, or a wall or ceiling of a room in which the recognition target is placed. The user environment may be specified by the recognition target itself, or the surface state or reflectance of objects present other than the recognition target. The user environment may be identified by the presence or absence of windows or blinds in a room in which the recognition target is placed when the recognition is performed. The user environment may be identified by the time series change in sunlight shining on the place in which the recognition target is placed when the recognition is performed.

[0030] The user environment estimation unit 326 outputs the estimation result of the user environment to the user environment object data generation unit 223 as user environment information. The user environment acquisition unit 325 may store the user environment information in the fourth data storage unit 334. The user environment object data generation unit 223 may generate the second image data based on the user environment information, as described above. The user environment object data generation unit 223 may acquire the user environment information from the user environment estimation unit 326 or from the fourth data storage unit 334. When the information used to estimate the user environment includes information capable of identifying the user environment, such as a lighting condition, the user environment estimation unit 326 may output the information capable of identifying the user environment itself as the user environment information.

[0031] The first control unit 120, the second control unit 220, and the third control unit 320 may be configured to include at least one processor to realize the functions of each component unit such as the standard environmental object data generation unit 121. The processor may execute a program to realize the functions of each component unit. The processor may be realized as a circuit that realizes the functions of each component unit. The processor may be realized as a circuit that collectively realizes the functions of multiple components units. The processor may be realized as a single integrated circuit. The integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as multiple integrated circuits and discrete circuits that are connected to each other in a communicable manner. The processor may be configured to include a CPU (Central Processing Unit). The processor may be configured to include a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The processor may be realized based on various other known technologies.

[0032] The first storage unit 130, the second storage unit 230, and the third storage unit 330 may be configured to include an electromagnetic storage medium such as a magnetic disk, or may be configured to include a memory such as a semiconductor memory or a magnetic memory. The first storage unit 130, the second storage unit 230, and the third storage unit 330 may be configured as a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The first storage unit 130, the second storage unit 230, and the third storage unit 330 may be configured to include an electromagnetic storage medium or memory corresponding to each component so as to hold data separately in each component such as the first data holding unit 131. The first storage unit 130, the second storage unit 230, and the third storage unit 330 may be configured to hold data of a plurality of components in one electromagnetic storage medium or memory. The first storage unit 130, the second storage unit 230, and the third storage unit 330 store various information and programs executed by the first control unit 120, the second control unit 220, and the third control unit 320, etc. The first storage unit 130, the second storage unit 230, and the third storage unit 330 may function as work memories for the first control unit 120, the second control unit 220, and the third control unit 320, respectively. The first control unit 120, the second control unit 220, and the third control unit 320 may be configured to include at least a portion of the first storage unit 130, the second storage unit 230, and the third storage unit 330.

[0033] <Imaging device 40> The imaging device 40 is configured to be capable of photographing the recognition target or an object other than the recognition target. The imaging device 40 may include an imaging element. The imaging device 40 may include an optical system including a lens, a mirror, or the like. The specifications of the imaging device 40 may be specified by resolution or sensitivity. The imaging device 40 may be configured to be capable of changing the resolution or sensitivity when photographing the recognition target or an object other than the recognition target. The specifications of the imaging device 40 may be specified by shutter speed or aperture. The imaging device 40 may be configured to be capable of changing the shutter speed or aperture when photographing the recognition target or an object other than the recognition target.

[0034] The first trained model generation device 110, the second trained model generation device 210, or the user environment estimation device 310 and the imaging device 40 may be configured to be able to communicate with each other via wired or wireless communication. The first trained model generation device 110, the second trained model generation device 210, the user environment estimation device 310, and the imaging device 40 may be equipped with a communication device. The communication device may be configured to be able to communicate using a communication method based on various communication standards. The communication device may be configured using known communication technology. Detailed description of the hardware and the like of the communication device is omitted. The functions of the communication device may be realized by one interface, or may be realized by separate interfaces for each connection destination.

[0035] (Example of operation of trained model generation system 1) The first trained model generation device 110 generates a first model by learning based on first image data including an image of a recognition target in a standard environment. The second trained model generation device 210 updates the first model by learning based on second image data including an image of a recognition target in a user environment to generate a second model, and outputs the second model as a trained model.

[0036] Below, a generation example of the first model and a generation example of the second model will be described.

[0037] <Example of generating the first model> The first control unit 120 of the first trained model generation device 110 generates first image data including an image of the recognition target in a standard environment. The standard environment is an environment for generating images that become teacher data used in learning to generate the first model. The first control unit 120 may acquire an image of the recognition target captured in the standard environment, and generate first image data including the acquired image. The first control unit 120 may generate an image that is assumed to be how the recognition target appears in the standard environment, and generate first image data including the generated image. The standard environment may be an environment that reduces the influence of at least shadows caused by the position of a light source on an image of the recognition target captured or an image that is assumed to be how the recognition target appears.

[0038] The standard environment is, for example, an environment in which a cup 50, which is an object to be recognized, is illuminated by a standard illuminator 41, as shown in FIG. 3 as a first environment 100. The standard illuminator 41 may be configured so as not to cast a shadow of the object to be recognized. The standard illuminator 41 may be configured, for example, to emit light that uniformly illuminates the object to be recognized from all directions. The standard illuminator 41 may be configured to include, for example, a panel-type lighting device. The standard illuminator 41 may be configured to include a plurality of lighting devices. The standard environment may be an actual environment, or may be a virtually set environment.

[0039] The first control unit 120 performs recognition learning in a standard environment using the first image data as training data, and generates a first trained model.

[0040] <Generation of the second model> The second control unit 220 of the second trained model generation device 210 generates second image data including an image of the recognition target in the user environment based on the user environment information. The user environment is an environment in which the recognition target is actually recognized using the trained model. The second control unit 220 generates an image that assumes how the recognition target will appear in the user environment, and generates second image data including the generated image.

[0041] The user environment is assumed to be an environment in which, as shown in FIG. 4 as the second environment 200, a cup 50 placed on a table 52 is photographed and recognized as a recognition target by the imaging device 40. In the example of FIG. 4, a shadow 50S of the cup 50 appears on the table 52. The second control unit 220 may acquire an image of the cup 50, the shadow 50S, and the table 52, that is, an image that specifies how the cup 50 appears in the user environment, and generate second image data including the acquired image. The second control unit 220 may acquire user environment information as described below, generate an image that assumes how the cup 50 appears in the user environment based on the user environment information, and generate second image data including the generated image.

[0042] The user lighting 42 may be configured to include, for example, a ring-shaped lighting device. The user lighting 42 may be configured to include various lighting devices. The user lighting 42 may be configured to include a plurality of lighting devices. The user environment may be a real environment or a virtually set environment.

[0043] The second control unit 220 updates the first model by learning the second image data as teacher data, and generates a second model. The second model generated by updating the first model can improve the recognition accuracy in the user environment. The second control unit 220 outputs the second model as a learned model.

[0044] <Estimation of user environment> The third control unit 320 of the user environment estimation device 310 may generate the user environment information by estimating it. The third control unit 320 can estimate the user environment information based on an image showing a cup 50 and a shadow 50S, which are the recognition targets, as shown in Fig. 4, for example. The third control unit 320 can also estimate the user environment information based on an image showing a marker 51 and a shadow 51S placed in the second environment 200, as shown in Fig. 5, for example. The marker 51 may include an object that is the recognition target, or may include an object that is not the recognition target.

[0045] The marker 51 has at least two visible surfaces. The marker 51 is arranged so that the angles of incidence of illumination light from the user illumination 42 on the two surfaces are different from each other. The marker 51 is arranged so that the two surfaces having different angles of incidence of illumination light are captured as one image by the imaging device 40. In addition, in FIG. 5, the imaging device 40 may include a first imaging device 40A and a second imaging device 40B. That is, the imaging device 40 may be configured to capture the marker 51 from two directions. The marker 51 may be arranged so that different surfaces of the marker 51 are captured in the images captured from each direction.

[0046] Specifically, the third control unit 320 estimates various conditions that identify the user environment based on an image of the marker 51. The third control unit 320 may estimate, for example, lighting conditions or specifications of the imaging device 40. The third control unit 320 may estimate information about objects that exist other than the recognition target, such as the table 52 on which the marker 51 is placed. The third control unit 320 generates or acquires the conditions that identify the user environment as user environment information.

[0047] The third control unit 320 may generate or acquire, as the user environment information, information that specifies a factor of noise occurring in the second image data in the user environment. The third control unit 320 may generate or acquire, as the user environment information, information that specifies a factor that generates a difference between the first image data and the second image data. The third control unit 320 may generate or acquire, as the user environment information, information on the position of a light source in the user environment, the intensity of light emitted from the light source, and the type of light source that specifies whether the light source is a point light source system or a scattered light system. The third control unit 320 may generate or acquire, as the user environment information, information on the optical properties of a table (e.g., table 52, etc.) on which the recognition target is placed, or a wall or ceiling of a room in which the recognition target is placed, in the user environment. The third control unit 320 may generate or acquire, as the user environment information, information on the shooting parameters of the shooting means used to recognize the recognition target in the user environment, or information on the vibration of the shooting means. The shooting means may include an imaging device 40.

[0048] <Example of procedure for generating trained models> The first control unit 120 of the first trained model generation device 110 and the second control unit 220 of the second trained model generation device 210 may execute a trained model generation method including the procedures of the flowcharts illustrated in Figures 6 and 7. The trained model generation method may be realized as a trained model generation program executed by a processor constituting the first control unit 120 and the second control unit 220. The trained model generation program may be stored in a non-transitory computer-readable medium.

[0049] The first control unit 120 and the second control unit 220 generate a trained model by executing the procedure of the flowchart exemplified in FIG. 6. The first control unit 120 generates first image data in a standard environment (step S1). The first control unit 120 generates a first model by a learning process in which the first image data is used as training data and the recognition target object shown in the first training data is used as teacher data (step S2). The second control unit 220 generates second image data in a user environment (step S3). The second control unit 220 updates the first model by a learning process in which the second image data is used as training data and the recognition target object shown in the second training data is used as teacher data, and generates a second model (step S4). The second control unit 220 outputs the second model as a trained model. After executing the procedure of step S4, the first control unit 120 and the second control unit 220 end the execution of the procedure of the flowchart in FIG. 6.

[0050] Furthermore, the third control unit 320 may execute the generation of the second image data in the procedure of step S3 in FIG. 6 based on the user environment information. The third control unit 320 may execute the procedure of the flowchart exemplified in FIG. 7 to generate user environment information and generate the second image data based on the user environment information. The third control unit 320 acquires the user environment data (step S11). The third control unit 320 generates the user environment information based on the user environment data (step S12). The third control unit 320 generates the second image data based on the user environment information (step S13). After executing the procedure of step S13, the third control unit 320 ends the execution of the procedure of the flowchart in FIG. 7 and proceeds to the procedure of step S4 in FIG. 6.

[0051] <Summary> As described above, the trained model generation system 1, the first trained model generation device 110, and the second trained model generation device 210 according to this embodiment generate a first model and a second model separately, and generate the second model by updating the first model based on user environment information. The first trained model generation device 110 and the second trained model generation device 210 can improve the robustness of the trained model generated as the second model by updating the first model based on user environment information to generate the second model.

[0052] In addition, the first trained model generation device 110 and the second trained model generation device 210 may generate a first model common to each user environment in a standard environment in order to generate a second model as a trained model applied to a plurality of user environments. After generating the first model, the first trained model generation device 110 and the second trained model generation device 210 can generate a second model by updating the first model through learning based on information on each user environment in order to generate a second model applied to each user environment. In other words, the first model is a common model for generating a second model corresponding to each of a plurality of user environments. By standardizing the learning for generating the first model, the computational load of learning for generating a second model applied to each user environment can be reduced. Furthermore, by standardizing the first model, the versatility of the first model can be increased.

[0053] The trained model generation system 1 may also have a third trained model generation device. A third model related to a user environment different from the second model may be generated. The third trained model generation device may have a configuration similar to that of the second trained model generation device 210. The third model may be generated in a similar manner to the second model. In order to generate trained models to be applied to each user environment, such as the second model and the third model, a first model common to each user environment may be generated in a standard environment. After generating the first model, the second trained model generation device 210 and the third trained model generation device can generate the second model and the third model by updating the first model through learning based on information about each user environment in order to generate the second model and the third model to be applied to each user environment. In other words, the first model is a common model for generating the second model and the third model corresponding to each of a plurality of user environments. By commonizing the learning for generating the first model, the computational load of learning for generating the second model and the third model to be applied to each user environment can be reduced. Furthermore, by standardizing the first model, the versatility of the first model can be increased. The third model may be generated, for example, each time. The third model does not have to be generated at the same time as the second model. Even if the second model is generated by updating the first model in order to generate the third model, the first model may be stored as the first model. Furthermore, the trained model generation system 1 may have trained model generation devices in the same number as the user environments, and may generate trained models in the same number as the user environments.

[0054] The trained model generation system 1 may also have a fourth trained model generation device. The fourth model may be generated based on the second model. For example, the control unit of the fourth trained model generation device may acquire a captured image of the user environment, and perform additional learning processing on the second model based on the acquired captured image in the user environment to generate the fourth model. Note that, unlike the second model, the fourth model may be generated by connecting an additional trained model that has undergone learning processing on the captured image in the user environment to the second model.

[0055] According to the trained model generation system 1, the first trained model generation device 110, and the second trained model generation device 210 of this embodiment, the versatility and robustness of the trained model can be ensured.

[0056] (Other embodiments) Other embodiments are described below.

[0057] <Example of how the object to be recognized appears> The lighting conditions in the user environment affect how objects in the user environment appear, such as the marker 51. Differences in how the marker 51 appears under different lighting conditions will be described below.

[0058] <<Example of how marker 51 looks when illuminated with parallel light>> As shown in Fig. 8, it is assumed that marker 51 is illuminated with parallel light whose light source is sunlight 43. Marker 51 is a triangular pyramid having a first surface 511, a second surface 512, a third surface 513, and a fourth surface 514. In Fig. 8, imaging device 40 is located on the front side of the paper and faces toward the back side of the paper to capture an image of marker 51.

[0059] The image shown in FIG. 9A shows a first surface 511 (see FIG. 8) and a second surface 512 (see FIG. 8) of the marker 51. The luminance of the first surface 511 located toward the lighting is higher than the luminance of the second surface 512. A shadow 51S of the marker 51 is cast on the table 52 by being illuminated with parallel light. The third control unit 320 of the user environment estimation device 310 may estimate the lighting conditions that illuminate the marker 51 based on the image shown in FIG. 9A. The third control unit 320 may estimate the lighting conditions based on the shape of the shadow 51S or the density of the shadow 51S. The third control unit 320 may estimate not only the lighting conditions but also information about the characteristics of the imaging device 40 or about objects other than the recognition target, such as the table 52 or the floor. The third control unit 320 may generate or acquire the estimation result as user environment information.

[0060] As an example of the marker 51 having another shape, Fig. 9B shows an image of a pyramidal marker 51. Fig. 9C shows an image of a quadrangular prism marker 51. The third control unit 320 may generate or acquire user environment information based on images of various markers 51.

[0061] <<Example of how marker 51 appears when illuminated by spotlight 44>> As shown in Fig. 10, it is assumed that only the periphery of marker 51 is illuminated with illumination light that spreads radially using spotlight 44 as a light source. Marker 51 is a triangular pyramid having first surface 511, second surface 512, third surface 513, and fourth surface 514. In Fig. 10, imaging device 40 is located on the front side of the paper and faces toward the back side of the paper to capture an image of marker 51.

[0062] The image shown in Fig. 11 shows a first surface 511 (see Fig. 10) and a second surface 512 (see Fig. 10) of the marker 51. The luminance of the first surface 511, which is located toward the lighting, is higher than the luminance of the second surface 512. As the marker 51 is illuminated with the illumination light, a shadow of the marker 51 is cast on the table 52. Furthermore, as only the periphery of the marker 51 is illuminated with radial illumination light, the table 52 is bright only in the vicinity of the marker 51. Furthermore, the shadow appears double due to diffraction of the illumination light.

[0063] The third control unit 320 may estimate the lighting conditions illuminating the marker 51 based on the image shown in Fig. 11. The third control unit 320 may estimate the lighting conditions based on the shape of the shadow of the marker 51 or the density of the shadow of the marker 51. The third control unit 320 may estimate not only the lighting conditions but also information on the characteristics of the imaging device 40 or on objects other than the recognition target, such as the table 52 or a floor. The third control unit 320 may generate or acquire the estimation result as user environment information.

[0064] <<Example of how marker 51 looks when illuminated from two directions by spotlights 44>> As shown in Fig. 12, it is assumed that a marker 51 is illuminated from two directions with illumination light having a first spotlight 44A and a second spotlight 44B as light sources. The marker 51 is a triangular pyramid having a first surface 511, a second surface 512, a third surface 513, and a fourth surface 514. In Fig. 12, the imaging device 40 is located on the front side of the paper and faces the back side of the paper to capture an image of the marker 51.

[0065] The image shown in Fig. 13 shows a first surface 511 (see Fig. 12) and a second surface 512 (see Fig. 12) of the marker 51. The luminance of the first surface 511 located toward the lighting is higher than the luminance of the second surface 512. When the marker 51 is illuminated with the lighting light, a shadow of the marker 51 is cast on the table 52 so as to extend in three directions. Specifically, a shadow corresponding to the lighting light from the first spotlight 44A, a shadow corresponding to the second spotlight 44B, and a shadow resulting from a combination of these two shadows are cast on the table 52.

[0066] The third control unit 320 may estimate the lighting conditions illuminating the marker 51 based on the image shown in Fig. 13. The third control unit 320 may estimate the lighting conditions based on the shape of the shadow of the marker 51 or the density of the shadow of the marker 51. The third control unit 320 may estimate not only the lighting conditions but also information on the characteristics of the imaging device 40 or on objects other than the recognition target, such as the table 52 or a floor. The third control unit 320 may generate or acquire the estimation result as user environment information.

[0067] <<Summary>> The third control unit 320 can estimate lighting conditions and the like in various user environments based on an image capturing the marker 51. The third control unit 320 can generate or acquire user environment information based on the estimation result. The marker 51 may be disposed so that at least two sides are captured by the imaging device 40. Furthermore, the imaging device 40 may be configured to capture images of the marker 51 from at least two directions.

[0068] <Extended environment information> The second control unit 220 of the second trained model generating device 210 or the third control unit 320 of the user environment estimation device 310 generates the second image data based on the user environment information as described above. The second control unit 220 or the third control unit 320 may generate information in which each parameter of the user environment information is changed within a predetermined range. The predetermined range may be set to a range in which the environment information changes in the time period in which the recognition by the second model is performed in the user environment, for example. Information in which at least one parameter among the multiple parameters of the user environment information is changed is also referred to as augmented environment information. The second control unit 220 or the third control unit 320 may generate multiple pieces of augmented environment information and generate second image data including an image assuming the appearance of the recognition target in each piece of augmented environment information. By learning an image assuming the appearance of the recognition target in the augmented environment information as teacher data, the robustness of the trained model may be improved.

[0069] <User Environment Estimation Device 310> The user environment estimation device 310 acquires image data obtained by capturing an image of a predetermined object in the user environment. The predetermined object may include the recognition target itself, or may include an object different from the recognition target, such as the marker 51. The user environment estimation device 310 may acquire the image data by an imaging means, or may acquire the image data from an external source. The user environment estimation device 310 estimates the user environment based on the image data.

[0070] The user environment estimation device 310 may estimate the user environment based on image data obtained by photographing a predetermined object from each of a plurality of directions. The user environment estimation device 310 may also estimate the user environment based on images obtained by photographing at least two of a plurality of faces of a predetermined object. The user environment estimation device 310 may also estimate the user environment based on images obtained by photographing two different faces of the predetermined object from at least two directions.

[0071] The ability of the user environment estimation device 310 to generate user environment information makes it easier to collect user environment information.

[0072] The functions of the user environment estimation device 310 may be realized as a user environment estimation method executed by the user environment estimation device 310. The functions of the user environment estimation device 310 may be realized as a user environment estimation program executed by a processor included in the user environment estimation device 310. For example, the user environment estimation program can estimate the user environment by comparing the user environment data with reference data indicating a predefined basic environment. The user environment estimation program and the reference data may be stored in the third data storage unit 333 or the fourth data storage unit 334.

[0073] <Examples of noise factors> As described above, the trained model generation system 1 according to the present embodiment generates a trained model that takes into account noise that occurs in the appearance of a recognition target in a user environment compared to the standard appearance of the recognition target. Hereinafter, a configuration for acquiring image data will be described as an example of a factor that causes noise.

[0074] In an image data acquisition configuration for acquiring an optical image of an object as digital data, an illumination light source strikes the object, and the reflected light is converted into a photoelectric signal by an optical sensor (image sensor, etc.) of a camera (image capture device 40, etc.), and the electrical signal is then converted into digital data, thereby acquiring image data. For this reason, the image data is subject to various optical or electrical variations and noises.

[0075] The noise of the image data includes noise caused by the camera. The noise caused by the camera includes, for example, color variation and noise due to the ISO sensitivity of the optical sensor, or brightness variation and noise. When taking a picture, the camera secures a dynamic range by changing the ISO sensitivity (amplification rate) of the optical sensor based on the input state of light. An increase in the sensitivity of the optical sensor can increase noise. The shutter speed and aperture of the camera are parameters that change the input state of light and are parameters related to the ISO sensitivity. These parameters can be easily referenced by being embedded in the image data as data in Exchangeable Image File Format (Exif). The noise caused by the camera also includes color reproducibility variation and noise due to the color reproduction range constraint of the optical sensor. The noise caused by the camera also includes distortion variation and noise in an optical system such as an optical lens, or peripheral light reduction variation and noise. The noise caused by the camera also includes noise based on the holding manner of the camera, for example, camera holding member and camera shake (vibration) noise due to the influence of surrounding vibrations.

[0076] Noise in image data includes noise caused by lighting. Noise caused by lighting includes, for example, shadow noise of an object associated with the lighting position (lighting coordinates). Noise caused by lighting includes contrast variation and noise of an object due to the type of light source (parallel light, scattered light, etc.), or shadow noise of an object. Noise caused by lighting includes contrast variation and noise of an object due to illuminance (brightness), or shadow noise of an object. Noise caused by lighting includes color shift variation and noise due to the color temperature of the lighting. Noise caused by lighting includes light and dark flicker variation and noise associated with the type of lighting or brightness adjustment, etc.

[0077] Noise in the image data includes noise caused by a work table such as the table 52. Noise caused by the work table includes noise caused by variations in reflectance due to the surface condition of the work table or noise caused by reflected light from the surface of the work table. Noise caused by the work table includes noise caused by poor separation from the object due to the color of the work table.

[0078] Noise in image data includes noise caused by the object, such as noise caused by variations in reflectance due to the surface condition of the object or noise caused by reflected light.

[0079] The second control unit 220 of the second trained model generation device 210 or the third control unit 320 of the user environment estimation device 310 may estimate each of the above-mentioned factors of noise in the image data based on the user environment data to generate user environment information. The second control unit 220 or the third control unit 320 may estimate some or all of the multiple factors of noise in the image data. In other words, the second control unit 220 or the third control unit 320 may estimate at least some of the multiple factors of noise in the image data. The noise caused by each of the above-mentioned factors greatly affects the recognition of the object. The trained model generation system 1 according to this embodiment can generate a trained model by learning using teacher data that takes these noises into account. As a result, the robustness of the trained model can be improved.

[0080] <Example of Marker 51 structure> As described above, the trained model generation system 1 according to the present embodiment can generate a trained model adapted to each user environment based on the user environment information. The user environment information can be generated based on an image of the marker 51 captured in the user environment. An example of the structure of the marker 51 will be described below.

[0081] When acquiring an object image as digital data, the user's image acquisition environment is subject to various optical and electrical variations and noises. Therefore, acquiring the user environment is necessary to increase robustness.

[0082] In order to acquire the user environment, for example, a marker 51 having the following three-dimensional structure may be used. The marker 51 may have a polyhedral structure and may have at least three faces. The marker 51 may have a structure that allows the shading of the shadows generated on each face to be distinguished when illuminated by lighting. Specifically, the marker 51 may have ridges that clearly define the boundaries between the faces. The marker 51 may have a structure that allows the reflectance of light on each face to be distinguished. The marker 51 may have a structure that allows the size of the marker 51 to be known, such as a mark indicating a specified size or a scale of dimensions. The marker 51 may have a lattice pattern or the like so that characteristics such as distortion of the optical system of the imaging means can be distinguished. The marker 51 may have a portion that has a known density, such as 18% of the grayscale. The marker 51 may have a portion that is a white point.

[0083] The marker 51 may be disposed so that at least two sides of the marker 51 are photographed by the photographing means. The marker 51 may be disposed so that the marker 51 is photographed from at least two or more different angles.

[0084] The second control unit 220 of the second trained model generating device 210 or the third control unit 320 of the user environment estimation device 310 may estimate, for example, the lighting position, the brightness, luminance or illuminance of the lighting, or the type of light source as the lighting conditions in the user environment based on the image data of the marker 51. The second control unit 220 or the third control unit 320 may also estimate the reflectance of the marker 51 or an object such as a workbench existing around the marker 51. The second control unit 220 or the third control unit 320 may estimate the lighting position based on the size and shadow of the marker 51. The second control unit 220 or the third control unit 320 may estimate the brightness of the lighting based on the density of the image of the marker 51 and the ISO sensitivity, shutter speed or aperture of the camera. The second control unit 220 or the third control unit 320 may estimate the contrast based on the image data of the edge portion of the marker 51 and the image data of the edge portion of the shadow, and may estimate the lighting conditions such as the type of the lighting light source (parallel light, scattered light, etc.). The second control unit 220 or the third control unit 320 may estimate the lighting conditions based on the distribution of pixel density of the edge portion of the marker 51 and the edge portion of the shadow. The second control unit 220 or the third control unit 320 may estimate the reflectance of the marker 51 based on a reflection image of the marker 51. The second control unit 220 or the third control unit 320 may estimate information about surrounding objects reflected in the marker 51 based on a reflection image of the marker 51. The second control unit 220 or the third control unit 320 may estimate the color temperature or spectrum of the lighting based on an image of the white point of the marker 51. The second control unit 220 or the third control unit 320 may estimate the distortion of the optical system of the imaging means based on an image of the grid pattern of the marker 51.

[0085] <Other configuration examples> A configuration that can be adopted as a feature of the trained model generation system 1 according to the present embodiment will be described below. The trained model generation system 1 may be configured to include a first recognition for recognizing an object in a standard environment and a second recognition for recognizing an object in a user environment in object recognition, and may be configured to enhance the recognition of the object in the first recognition by recognizing the object in the first recognition and enhance the robustness of the recognition in the user environment in the second recognition by recognizing the object in the second recognition after having the object recognized in the first recognition. In addition, the trained model generation system 1 may store at least an object recognition algorithm, or an object recognition algorithm and an object data set in the first recognition. In addition, in the trained model generation system 1, the standard environment object data generation means for the first recognition may be configured with a light and a light holding member for holding the light, an object and a member for holding the object, and an image conversion system for converting the object into data. In addition, the light of the standard environment object data generation means for the first recognition may be configured with two or more lights. The lighting of the standard environment object data generating means for the first recognition may be configured to adjust the color temperature of the lighting. The image conversion system for converting the object of the standard environment object data generating means for the first recognition into data may be configured to generate data based on a two-dimensional color image or a three-dimensional color image and distance data. The standard environment object data generating means for the first recognition may be configured to be a virtual environment. In the second recognition, the learning of object recognition is sufficiently performed in the first recognition, and the stored object recognition algorithm, or the object recognition algorithm and object data set, may be copied to the beginning of the recognition learning of the second recognition. The second recognition user environment object data generating means may be configured to configure a user environment based on the result of estimating the user environment and perform recognition. The user environment based on the result of estimating the user environment by the second recognition user environment object data generating means may be configured to include lighting, a lighting holding member for holding the lighting, an object, and a member for holding the object, as well as an image conversion system for converting the object into data. Furthermore, the illumination of the user environment object data generating means for the second recognition may be configured with two or more illuminations.Furthermore, the lighting of the user environment object data generating means for the second recognition may be configured to be able to adjust the color temperature of the lighting. Furthermore, the image conversion system for digitizing the object of the user environment object data generating means for the second recognition may be configured to generate data based on a two-dimensional color image or a three-dimensional color image and distance data. Furthermore, in the second recognition, the object may be recognized by constructing a virtual user environment based on the result of estimating the user environment.

[0086] In addition, in the first recognition and the second recognition, the standard environment and the user environment may have common environmental elements. That is, for example, when generating the first image data taking into account the standard environment, if lighting is also generated as an environmental element, when generating the second image data taking into account the user environment, lighting may also be generated as an environmental element. Note that the data indicating the standard environment and the user environment may have the same data type. In this case, for example, the standard environment or the user environment can be used for the same software.

[0087] The user environment may include a means for measuring the user environment and a means for estimating the user environment from information obtained therefrom. The means for measuring the user environment may be configured to have a three-dimensional object and acquire user environment data such as physical information (size, density, reflection) of the three-dimensional object and image data of the three-dimensional object from two or more different angles. Also, the environment estimation means may be configured to estimate user environment information such as lighting position, number of lights, brightness, light source type, or reflectance from the user environment data. Also, the means for estimating the user environment from the user environment data may be configured to geometrically estimate the user environment from two image data. Also, the three-dimensional object for measuring the user environment may have a white object on the three-dimensional object. A white object may be placed near the three-dimensional object. Also, the three-dimensional object for measuring the user environment may have a grayscale density object on the three-dimensional object. A grayscale density object may be placed near the three-dimensional object. The reflectance of the grayscale density object may be 18%. The image data of the three-dimensional object may include a two-dimensional color image or a three-dimensional color image and distance data.

[0088] The trained model generation system 1 may be configured to save or accumulate user environment data. The trained model generation system 1 may be configured to save or accumulate user environment information.

[0089] In the trained model generation system 1, the user environment object recognition unit 224 and the standard environment object recognition unit 122 may be configured identically or similarly. Even if the user environment object recognition unit 224 and the standard environment object recognition unit 122 are the same, the learning result may differ depending on the input data of the standard environment or the user environment.

[0090] The first trained model generation device 110 and the second trained model generation device 210 may be configured as the same device. The timing at which the first trained model generation device 110 performs the first learning to generate the first model and the timing at which the second trained model generation device 210 performs the second learning to generate the second model may be different from each other.

[0091] The standard environment object data generating unit 121 and the user environment object data generating unit 223 may be configured as a common object data generating unit. The object data generating unit functioning as the standard environment object data generating unit 121 generates standard environment object data by reading standard environment information. The object data generating unit functioning as the user environment object data generating unit 223 generates user environment object data by reading user environment information.

[0092] The function of the user environment estimation unit 326 of the third control unit 320 of the user environment estimation device 310 may be realized by the second control unit 220 of the second trained model generation device 210. In this case, the function of the fourth data holding unit 334 of the third storage unit 330 of the user environment estimation device 310 is realized by the second storage unit 230 of the second trained model generation device 210. Here, it is assumed that the second trained model generation device 210 is owned by a vendor that provides a trained model. Also, it is assumed that the user environment estimation device 310 is owned by a user that performs recognition using the trained model. In this case, the function of the user environment estimation unit 326 is realized by the second trained model generation device 210, so that the user environment is estimated on the vendor side. In other words, it is not necessary for the user environment to be estimated on the user side. User convenience is improved.

[0093] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions, etc. included in each component, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, etc. can be combined into one or divided.

[0094] All of the features described in this disclosure and / or all steps of any method or process disclosed herein may be combined in any combination except those combinations in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.

[0095] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.

[0096] A part of the method according to the present disclosure may be manually executed by a human. For example, an instruction to start the work of generating a learning model may be manually executed. Also, a folder in which a learning dataset is saved may be manually specified. The trained model generation system 1 according to the present disclosure, a trained model generation device such as the first trained model generation device 110 or the second trained model generation device 210, or a user environment estimation device 310 may be configured to accept an input regarding the content that a human intends to execute manually. For example, the trained model generation system 1 may be communicatively connected to the trained model generation device or the user environment estimation device 310, and may have an input device that accepts user input. Also, the trained model generation device or the user environment estimation device 310 may have an input unit that accepts user input. As a result, the trained model generation system 1, for example, can accept an input of a user's work start instruction or a user input that specifies a storage destination of the learning data during the learning process. The input device or the input unit may be configured to include, for example, a touch panel or a touch sensor, or a pointing device such as a mouse. The input device or input unit may be configured to include physical keys, or may be configured to include a voice input device such as a microphone.

[0097] The present disclosure may also be realized as a trained model generation device. Specifically, the trained model generation device in one embodiment includes: (1) A control unit is provided, The control unit is acquiring a first model by performing a learning process on the estimation target using first image data indicating the estimation target in a first environment as learning data; acquiring second image data showing the estimation target in a second environment in which the estimation is performed; generating a second model based on the first model using the second image data as learning data; Outputting a trained model based on the second model; The second image data includes an image that assumes how the estimation target will appear in the second environment, based on user environment information about the second environment.

[0098] (2) In the trained model generation device of (1) above, the first environment may be an environment that reduces at least the effect of shadows cast due to the position of a light source on an image captured of the estimation target or an image that assumes the appearance of the estimation target.

[0099] (3) In the trained model generation device of (1) or (2) above, the control unit may acquire, as the user environment information, information that identifies a cause of noise that occurs in the second image data in the second environment.

[0100] (4) In any one of the trained model generation devices (1) to (3) above, the control unit may acquire, as the user environment information, information that identifies a factor that causes a difference between the first image data and the second image data.

[0101] (5) In any one of the trained model generation devices (1) to (4) above, the control unit may acquire, as the user environment information, information on a position of a light source in the second environment, an intensity of light emitted from the light source, and a type of light source specifying whether the light source is a point light source system or a scattered light system.

[0102] (6) In any one of the trained model generation devices (1) to (5) above, the control unit may acquire, as the user environment information, information on optical properties of a stand on which the estimation target is placed in the second environment.

[0103] (7) In any one of the trained model generation devices (1) to (6) above, the control unit may acquire, as the user environment information, information on shooting parameters of a shooting means used to recognize the estimation target in the second environment.

[0104] (8) In any one of the trained model generation devices (1) to (7) above, the control unit may acquire, as the user environment information, information regarding vibration of an imaging means used to recognize the estimation target in the second environment.

[0105] (9) In any one of the trained model generation devices (1) to (8) above, the control unit may generate a plurality of augmented environmental information in which each parameter of the user environment information is changed within a predetermined range, and generate the second image data for each of the augmented environmental information.

[0106] The present disclosure may also be realized as a user environment estimation device. Specifically, the user environment estimation device in one embodiment includes: (10) A control unit that estimates a user environment from which data to be estimated is acquired; The control unit outputs a result of estimating the user environment based on image data obtained by photographing a predetermined object in the user environment as user environment information regarding the user environment.

[0107] (11) In the user environment estimation device of (10) above, the control unit may estimate the user environment based on image data capturing an object other than the estimation target as the specified object in the user environment.

[0108] (12) In the user environment estimation device according to (10) or (11) above, the control unit may estimate the user environment based on image data obtained by photographing the estimation target as the specified object in the user environment.

[0109] (13) In the user environment estimation device according to any one of (10) to (12) above, the control unit may estimate the user environment based on image data obtained by photographing the specified object from each of a plurality of directions.

[0110] (14) In any one of the user environment estimation devices (10) to (13) above, the image data may include images of at least two of a plurality of surfaces of the specified object, or images of two different surfaces of the specified object photographed from at least two directions.

[0111] The present disclosure may also be realized as a trained model generation program. (15) acquiring a first model by performing a learning process on the estimation target using first image data indicating the estimation target in the first environment as learning data; acquiring second image data showing the estimation target in a second environment in which the estimation is performed; generating a second model based on the first model using the second image data as training data; Outputting a trained model based on the second model. The trained model generating device executes the above. The second image data includes an image that assumes how the estimation target will appear in the second environment, based on user environment information about the second environment.

[0112] The present disclosure may also be realized as a user environment estimation program. Specifically, the user environment estimation program in one embodiment includes: (16) Outputting a result of estimating the user environment based on image data obtained by photographing a predetermined object in the user environment as user environment information regarding the user environment.

[0113] The present disclosure can also be realized as a trained model generation system. Specifically, the trained model generation system in one embodiment includes: (17) A trained model generation device according to any one of (1) to (9) above, and a user environment estimation device according to any one of (10) to (14) above. The trained model generation device acquires, from the user environment estimation device, user environment information estimated by the user environment estimation device. [Explanation of symbols]

[0114] 1. Trained model generation system 40 Imaging device (40A: first imaging device, 40B: second imaging device) 41 Standard Lighting 42 User Lighting 43. Sunlight 44 Spotlight (44A: 1st spotlight, 44B: 2nd spotlight) 50 Cup (50S: Shadow of a Cup) 51 Marker (51S: Marker's Shadow, 511~514: 1st~4th sides) 52 Table 100 1st environment (standard environment) 110 first trained model generating device (120: first control unit, 121: standard environment object data generating unit, 122: standard environment object recognizing unit, 130: first storage unit, 131: first data holding unit) 200 Second environment (user environment) 210 second trained model generating device (220: second control unit, 223: user environment object data generating unit, 224: user environment object recognition unit, 230: second storage unit, 232: second data holding unit) 310 User environment estimation device (320: third control unit, 325: user environment acquisition unit, 326: user environment estimation unit, 330: third storage unit, 333: third data holding unit, 334: fourth data holding unit)

Claims

[Claim 1] A trained model generation method, acquiring a first model obtained by performing a learning process on the estimation target using first image data representing the estimation target in a first environment as learning data; acquiring second image data showing the estimation target in a second environment in which estimation is performed; generating a second model based on the first model using the second image data as training data; outputting a trained model based on the second model; Including, the second image data includes an image that is an image of how the estimation target will appear in the second environment, based on user environment information about the second environment; How to generate a trained model.

Images