Information processing device, information processing method and program

Abstract

To provide an information processing device, an information processing method and a program capable of, stably at high speed, detecting the region of an object in a picked-up image even in a case in which the color of the object changes in accordance with optical conditions.SOLUTION: An information processing device 100 includes: a color region detecting unit 104 that repeatedly executes a first process of detecting a certain object from a real space using setting information stored in a data storing unit 103; a region detecting unit 108 that executes a second process which is a process of detecting the certain object from the real space and which is a process requiring a longer time than the time required for the first process; a color updating unit 109 that updates the setting information based on the result of the second process; and a control unit 110 that executes a control so as to execute a predetermined process based on the result of the first process.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, research has been conducted on mixed reality (MR), which overlays virtual space information onto real space in real time and presents it to users. Mixed reality is a technology that displays a synthetic image by overlaying an image of a virtual space corresponding to the position and orientation of an imaging device such as a head mounted display (HMD) onto all or part of a real image captured by the imaging device.

[0003] An MR system using mixed reality technology can express the sense of distance between objects by not displaying the area of ​​the object located at the back in the area where the real object and the virtual object overlap in the synthetic image. For example, when a user wearing an HMD holds a real object such as his / her hand or a tool in front of a virtual object, the MR system can display the hand or tool as if it is in front of the virtual object by not drawing a virtual object in the area of ​​the hand and tool on the captured image. This makes it easier to grasp the positional relationship between the virtual object and the real object, and the user can easily verify the work using the real hand and tool in the virtual space.

[0004] The MR system detects a real object and measures the distance from the user to the real object, thereby expressing the positional relationship between the real object and the virtual object. Patent Document 1 describes a technology that generates polygons of fingers from the contour of a skin-colored area extracted from a captured image, determines a collision between a real object and a virtual object, and draws the fingers so that they do not sink into the virtual object. Non-Patent Document 1 describes a technology that extracts the area of ​​the hand using a trained model that has been trained with various hand images using a deep neural network. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-022292 A [Non-patent literature]

[0006] [Non-Patent Document 1] Zhao, Hengshuang, et al. “ICnet for Real-Time Semantic Segmentation on High-Resolution Images.” Proceedings of the European conference on computer vision (ECCV). 2018. Summary of the Invention [Problem to be solved by the invention]

[0007] However, because the color of fingers differs from user to user and also changes depending on optical conditions such as the surrounding lighting, even if the skin-colored area is extracted, the finger area may not be extracted correctly.In addition, since the MR system generates a synthetic image to be displayed for each frame, if the object area is extracted for each frame using a trained model based on a neural network, the processing time becomes long, making real-time processing difficult.

[0008] An object of the present invention is to provide an information processing device that can quickly and stably detect an object region in a captured image even when the color of the object changes depending on optical conditions. [Means for solving the problem]

[0009] The information processing device of the present invention is characterized in having a first detection means for repeatedly performing a first process for detecting a specific object from real space using setting information stored in a memory unit, a second detection means for performing a second process for detecting the specific object from the real space, which is a process that takes a longer time than the time required for the first process, an update means for updating the setting information based on a result of the second process, and a control means for controlling the performance of a predetermined process based on the result of the first process. Effect of the Invention

[0010] According to the present invention, even if the color of the object changes depending on the optical conditions, the area of ​​the object in the captured image can be detected quickly and stably. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram illustrating a functional configuration of the information processing system. [Diagram 2] FIG. 1 is a schematic diagram illustrating a device configuration of an information processing system. [Diagram 3] 4 is a flowchart illustrating a display image generating process according to the first embodiment. [Figure 4] 10 is a flowchart illustrating a color information update process according to the second embodiment. [Diagram 5] FIG. 11 is a diagram illustrating a color histogram used in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment 1> Hereinafter, a first embodiment for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a functional configuration of an information processing system according to the first embodiment. The information processing system includes an imaging device 120, an input device 130, a display device 140, and an information processing device 100. The information processing device 100 is connected to the imaging device 120, the input device 130, and the display device 140. The information processing device 100 may be configured as an integral part of at least one of the imaging device 120, the input device 130, and the display device 140, without being limited to the example of FIG. 1.

[0013] The information processing device 100 includes a captured image acquisition unit 101, a color registration unit 102, a data storage unit 103, a color region detection unit 104, a depth acquisition unit 105, a model generation unit 106, a display image generation unit 107, a region detection unit 108, and a color update unit 109. The information processing device 100 also includes a control unit 110 that controls the processing of each processing unit of the information processing device 100.

[0014] The captured image acquisition unit 101 stores, in a data storage unit 103 (storage unit), captured images of real space captured by the imaging device 120. The imaging device 120 is, for example, a stereo camera equipped with a plurality of cameras, and is capable of capturing stereo images.

[0015] The color registration unit 102 generates a color information table based on the captured image acquired from the captured image acquisition unit 101 and the user's input information acquired from the input device 130. The color registration unit 102 registers the color of the object to be detected, designated by the user on the captured image, in the color information table as a detected color. For example, the user can designate the color of the object to be registered by surrounding a part of the area of ​​the object to be detected with an ellipse or a rectangle on the captured image using the input device 130 such as a mouse. The color registration unit 102 registers a plurality of colors, such as a color included in the designated area and a color near the designated color on the color space, as detected colors. This method is not limited to the above, and the color registration unit 102 may register the detected color in the color information table by a known technique.

[0016] The color information registered in the color information table is expressed in a color space such as RGB, YUV, HSV, Lab, etc. The color information is not limited to chromatic color information, but may be achromatic color information (luminance information). The color information may be information extracted from an infrared image captured by an infrared sensor.

[0017] When there are multiple objects to be detected, the color registration unit 102 registers the color of each object as a detection color in the color information table based on the user's designation. In this case, the color registration unit 102 may register the detection color in association with the object to be detected, or may register the detection color without association with the object to be detected. When the detection color and the object are registered in association with each other, the color region detection unit 104 can detect the multiple objects by distinguishing them.

[0018] Furthermore, the color registration unit 102 may register, in the color information table, a color that is not to be detected as the color of an object, as a non-detection color, in response to a user's designation. The non-detection color is used, for example, in a case where a color similar to a detection target, such as a user's hand, is included in the background image, so that a background of a color similar to the detection target is not detected. The color registration unit 102 registers, in the color information table, a color designated by the user as a color similar to the detection target on the captured image of the background, as a non-detection color.

[0019] Furthermore, when a user specifies a detection color, if the specified color causes an area other than the object to be detected, the color registration unit 102 may register the color of the area other than the object as a non-detection color. By excluding areas of non-detection colors from detection targets, detection of areas other than the object that are not intended by the user is reduced, and the color area detection unit 104 can detect the object with high accuracy. The color registration unit 102 stores the generated color information table in the data storage unit 103.

[0020] The color region detection unit 104 (first detection means) detects a specific object (real object) to be modeled from a captured image of real space stored in the data storage unit 103, using color information registered in a color information table. The color region detection unit 104 generates a color region detection image indicating the area of ​​the detected object. The color region detection unit 104 repeatedly performs a process of detecting a specific object from a captured image (frame) acquired by the captured image acquisition unit 101 and stored in the data storage unit 103. The color information such as detected colors and non-detected colors registered in the color information table is an example of setting information. The process of detecting a specific object from real space by the color region detection unit 104 using color information is an example of a first process.

[0021] The depth acquisition unit 105 acquires depth information based on the captured image and generates a depth image. The depth acquisition unit 105 can obtain the depth of the entire image from the stereo image by, for example, a method such as SemiGlobalMatching (SGM). The depth acquisition unit 105 can also limit the area for which the depth is to be obtained and improve accuracy by masking areas of the captured image other than the area of ​​the object to be detected using the color region detection image generated by the color region detection unit 104.

[0022] The model generating unit 106 detects a detection target object (a real object such as a hand) using the color region detection image generated by the color region detection unit 104 and the depth image generated by the depth acquisition unit 105, and generates a three-dimensional model. For example, the model generating unit 106 can generate a two-dimensional polygon from the color region detection image and generate a three-dimensional polygon (three-dimensional model) of the detected object using depth information of the depth image. The model generating unit 106 may generate a two-dimensional polygon from the color region detection image using a MarchingSquares method or the like, or may generate a two-dimensional polygon from the contour of the object extracted in the color region detection image using a Delaunay triangulation method. In addition, the model generating unit 106 may generate a three-dimensional model using model fitting or the like when the shape of the detection target object is limited to a hand or the like that is known in advance.

[0023] The display image generating unit 107 synthesizes a Computer Graphics (CG) model (virtual object) stored in the data storage unit 103 with a captured image to generate a display image to be displayed on the display device 140. The display image generating unit 107 first draws the captured image as a background, then enables a depth test and generates a transparent thickness model (a three-dimensional model of the object generated by the model generating unit 106). By performing such processing, the display image generating unit 107 can generate a display image in which the 3D model of the real object and the CG model are appropriately expressed and these models are superimposed on the captured image.

[0024] The area detection unit 108 (second detection means) detects the object to be detected, such as a hand or a face, with higher accuracy from the captured image stored in the data storage unit 103. For example, a trained model that is trained to be able to detect the object to be detected using a deep neural network may be generated in advance, and the area detection unit 108 may detect the object to be detected using the trained model generated. The area detection unit 108 may also obtain a rectangular area including the object to be detected using a cascade classifier or the like, and perform contour fitting using the Snakes method. The area detection unit 108 may detect the object to be detected by a method of modeling a non-rigid object, such as ActiveShapeModels and ActiveAppearanceModel, and fitting using contour information and texture information.

[0025] The process of detecting an object by the area detection unit 108 takes longer than the process of detecting an object of a detection color registered in the color information table by the color area detection unit 104, but the object to be detected can be detected with high accuracy even if there is no color information (setting information) registered in advance. The area detection unit 108 generates an object area detection image that shows the area of ​​the detected object.

[0026] The color update unit 109 updates the color information table based on color information extracted from the object region included in the object region detection image generated by the region detection unit 108. The color update unit 109 does not register as a detected color a color that is registered as a non-detected color in the color information table by the user, even if the color is extracted from the object region of the object region detection image. The color update unit 109 records the updated color information table in the data storage unit 103.

[0027] Fig. 2 is a schematic diagram illustrating an example of the device configuration of the information processing system according to embodiment 1. In the example of Fig. 2, an information processing device 100 is connected to an imaging device 120, a display device 140, and an input device .

[0028] The imaging device 120 is, for example, a camera incorporated in the HMD, and captures an image of a subject to be modeled (an object to be detected). The HMD may be a video see-through type or an optical see-through type. The display device 140 includes a display such as an HMD display and a monitor of a personal computer (PC). The input device 130 includes, for example, a mouse and a keyboard connected to a PC.

[0029] 3(a) to 3(c) are flowcharts illustrating a display image generation process according to the first embodiment. Each of the flowcharts in FIG. 3(a) to 3(c) is executed asynchronously in a respective thread. The information processing device 100 generates a display image using polygons of an object detected using color information, and executes a process of updating the detected colors registered in the color information table asynchronously with the generation of the display image.

[0030] The information processing device 100 detects an object with higher accuracy than area extraction using color information, and dynamically updates the color information table with the color extracted from the detected object, so that the area of ​​the object can be stably extracted even if the color of the object to be detected changes due to a change in optical conditions. The information processing device 100 generates a display image at a higher speed by detecting an object using color information in the color information table, and realizes faster and more stable image display by detecting an object with high accuracy and updating the color information in parallel with the generation process of the display image. The information processing device 100 can suppress the user's discomfort and visually induced motion sickness by detecting an object at a higher speed and generating a display image (for example, a frame rate of 60 fps or more).

[0031] 3A shows a flowchart of a display image generation process executed by the control unit 110 on the main thread when the captured image acquisition unit 101 acquires a captured image. The display image generation process is repeatedly executed for each captured image (frame) acquired by the captured image acquisition unit 101.

[0032] In step S311, the control unit 110 instructs the generation of polygons (three-dimensional models of objects to be detected) on the polygon generation thread. The control unit 110 executes the polygon generation process shown in Fig. 3(b) as asynchronous processing with the processing shown in Fig. 3(a). After instructing the generation of polygons, the control unit 110 advances the processing to step S312.

[0033] In step S312, the control unit 110 instructs updating of color information on the color information update thread. The control unit 110 executes the color information update process shown in Fig. 3(c) as asynchronous processing with the processing shown in Fig. 3(a). After instructing to update the color information, the control unit 110 advances the processing to step S313.

[0034] In step S313, the control unit 110 waits for the generation of polygons to be completed by the process shown in Fig. 3(b). When the control unit 110 receives a notification that the generation of polygons is completed, the control unit 110 advances the process to step S314. In step S314, the display image generation unit 107 generates a display image using the polygons (three-dimensional models of the object to be detected) generated by the process shown in Fig. 3(b).

[0035] 3B shows a flowchart of the polygon generation process executed on the polygon generation thread when a command to generate a polygon is issued in step S311. In step S321, the color region detection unit 104 detects a region of the detection color registered in the color information table (first process) and generates a color region detection image. In step S322, the depth acquisition unit 105 acquires depth information and generates a depth image.

[0036] In step S323, the model generation unit 106 generates a 3D model of the detection target object from the color region detection image generated in step S321 and the depth image generated in step S322. In step S324, the model generation unit 106 notifies the control unit 110, which executes processing on the main thread, that the generation of polygons has been completed.

[0037] Fig. 3(c) shows a flowchart of the color information update process executed on the color information update thread when an instruction to update color information is given in step S312. In step S331, the area detection unit 108 determines whether or not the color information update process shown in Fig. 3(c) is being executed. If the color information update process is being executed, the process shown in Fig. 3(c) ends. If the color information update process is not being executed, the control unit 110 advances the process to step S332.

[0038] In step S332, the area detection unit 108 detects the area of ​​the detection target object by a trained model trained to detect a specific object using a deep neural network or the like, without using a color information table. The process of detecting the area of ​​the object in step S332 is executed asynchronously with the generation of the display image in order to detect the area of ​​the object more accurately and take more time than the detection process in step S321.

[0039] In step S333, color update unit 109 extracts the color contained in the area of ​​the object detected in step S332, and updates the detected color registered in the color information table.

[0040] In the above-described first embodiment, the information processing device 100 repeatedly performs a process (first process) of detecting an object whose detection color is registered in the color information table from a captured image in real space. In addition, the information processing device 100 performs a process of learning using a deep neural network or the like asynchronously with the first process. The information processing device 100 performs a process (second process) of detecting a detection target object from a captured image in real space using the trained model. The information processing device 100 extracts the color of the area of ​​the object detected by the second process and updates the information of the detected color registered in the color information table. Since the color information (setting information) in the color information table is dynamically updated by the second process, the information processing device 100 can stably detect the object by the first process. That is, even if the color of the object to be detected changes from the detection color registered in advance due to a change in optical conditions, the information processing device 100 can extract the area of ​​the object quickly and stably because the detection color is appropriately updated by the second process. Therefore, the information processing device 100 can quickly and stably execute a process (an example of a predetermined process) of generating a display image by synthesizing a model and a CG model of the object detected based on the result of the first process with a captured image in real space.

[0041] <Embodiment 2> In the first embodiment, the area detection unit 108 executes the object detection process (second process) regardless of whether the object to be detected is present in the captured image. In the second embodiment, the information processing device 100 reduces calculation resources and power consumption by not executing the object detection process by the area detection unit 108 for frames in which the object to be detected is not present in the captured image. The display image generation process and polygon generation process according to the second embodiment are similar to the processes in Figs. 3(a) and 3(b) described in the first embodiment.

[0042] Fig. 4 is a flowchart illustrating a color information update process according to the second embodiment. The process shown in Fig. 4 is executed by the area detection unit 108 and the color update unit 109 of the information processing device 100. In the second embodiment, when the control unit 110 instructs updating of color information in step S312 in Fig. 3(a), the area detection unit 108 and the color update unit 109 execute the color information update process shown in Fig. 4 instead of the process shown in Fig. 3(c).

[0043] In step S401, the area detection unit 108 determines whether or not the color information update process shown in Fig. 4 is being executed. If the color information update process is being executed, the process shown in Fig. 4 ends. If the color information update process is not being executed, the control unit 110 proceeds to step S402.

[0044] In step S402, the area detection unit 108 acquires a captured image from the data storage unit 103. In step S403, the area detection unit 108 determines whether or not a reference histogram exists in the data storage unit 103. The reference histogram is a color histogram of a captured image when a detection target object does not exist in the captured image. The reference histogram is registered in the data storage unit 103 when a detection target object was not detected in the previous frame. If the reference histogram exists, the process proceeds to step S404. If the reference histogram does not exist, the process proceeds to step S406.

[0045] The reference histogram is a color histogram of the entire captured image. The color histogram is a count of the frequency of occurrence of colors in the captured image, and may be calculated for each color in a predetermined color space or for each channel. The reference histogram is used to determine how much the current frame has changed from the previous frame. The reference histogram is an example of a reference color distribution, and the color histogram is an example of a color distribution.

[0046] In step S404, the area detection unit 108 generates a color histogram from the captured image. In step S405, the area detection unit 108 determines whether the difference between the generated color histogram and the reference histogram is smaller than a predetermined threshold. The difference between the two color histograms may be, for example, the sum of the differences in frequency for each color in the color histograms, or the most frequent value of the differences in frequency for each color in the color histograms. If the difference between the two color histograms is equal to or greater than the predetermined threshold, the process proceeds to step S406. If the difference between the two color histograms is smaller than the predetermined threshold, the process shown in FIG. 4 ends.

[0047] In step S406, the area detection unit 108 detects the area of ​​the object to be detected using a trained model trained by a deep neural network or the like without using a color information table (second process). In S407, the area detection unit 108 determines whether the object to be detected exists in the captured image. If the object to be detected exists in the captured image, the process proceeds to step S408. If the object to be detected does not exist in the captured image, the process proceeds to step S410.

[0048] In step S408, the color update unit 109 extracts the color contained in the area of ​​the object detected in step S406, and updates the color information table with the extracted color as the detected color of the object. If the extracted color is a non-detected color, the color update unit 109 does not register the extracted color as the detected color in the color information table. In step S409, the color update unit 109 deletes the reference histogram from the data storage unit 103, and ends the process.

[0049] In step S410, the area detection unit 108 generates a color histogram of the captured image. In step S411, the area detection unit 108 registers the generated color histogram in the data storage unit 103 as a reference histogram.

[0050] Figures 5(a)-5(c) show example color histograms for a possible scene when the object to be modeled is a user's hand. The example color histograms in Figures 5(a)-5(c) show superimposed histograms of U and V color values ​​in the YUV color space.

[0051] 5(a) and 5(b) show scene 501 and scene 502, respectively, in which no hands are shown, and their respective color histograms. Fig. 5(c) shows scene 503, which is substantially the same as scene 501 and scene 502, in which a hand is shown, and its color histogram.

[0052] The scenes are assumed to have been captured in the order of scene 501, scene 502, and scene 503. Because no hand is captured in scene 501, area detection unit 108 determines in step S407 that no object to be detected is present, and registers color histogram 511 generated in step S410 in data storage unit 103 as a reference histogram.

[0053] Scene 502 is captured when the position and orientation of imaging device 120 is different from that of scene 501, but like scene 501, no hands are captured in scene 502. Because color histogram 511 of scene 501 is registered in data storage unit 103 as a reference histogram, area detection unit 108 generates color histogram 512 for scene 502 in step S404. Area detection unit 108 compares color histogram 512 with the reference histogram (color histogram 511) in step S405. It is determined that the difference between color histogram 511 and color histogram 512 is equal to or smaller than a threshold, and the color information update process in FIG. 4 ends.

[0054] Scene 503 is captured with the image capturing device 120 in substantially the same position and orientation as scene 502. However, unlike scenes 501 and 502, scene 503 includes a hand, which is the detection target. Since color histogram 511 of scene 501 is not deleted but is registered in data storage unit 103 as a reference histogram, area detection unit 108 generates color histogram 513 of scene 503 in step S404. Area detection unit 108 compares color histogram 513 with the reference histogram (color histogram 511) in step S405. Color histogram 511 and histogram 513 have different frequencies of the most frequent values, and histogram 513 shows a peak that does not exist in histogram 511 and histogram 512. It is determined that the difference between color histogram 511 and color histogram 513 is greater than the threshold, and in step S406, region detection unit 108 Execute the area detection process.

[0055] In the above-mentioned second embodiment, the information processing device 100 judges whether or not a detection target object exists in a captured image from the difference between a color histogram (reference histogram) of a captured image in which the detection target object does not exist and a color histogram of a captured image (frame) currently being processed. When the detection target object does not exist in a captured image, the information processing device 100 does not execute object detection by the area detection unit 108, thereby reducing calculation resources and power consumption.

[0056] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments and includes various forms within the scope of the gist of the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and various forms can be appropriately combined.

[0057] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0058] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first detection means for repeatedly performing a first process for detecting a specific object from a real space by using setting information stored in a storage unit; a second detection means for performing a second process that detects the specific object from the real space and that takes a longer time than the first process; an update means for updating the setting information based on a result of the second process; A control means for controlling the device to perform a predetermined process based on a result of the first process; 13. An information processing device comprising: (Configuration 2) The predetermined process is a process of generating a display image by synthesizing a virtual object with a captured image of the real space. 2. The information processing device according to configuration 1. (Configuration 3) the setting information includes color information of the specific object; the first detection means detects the specific object using color information of the specific object; The update means updates the setting information based on color information of the specific object detected in the second process. 3. The information processing device according to configuration 1 or 2. (Configuration 4) the setting information includes information on a detection color to be detected as the color of the specific object and information on a non-detection color not to be detected as the color of the specific object, When the color extracted from the specific object detected in the second process is the non-detected color, the update means does not register the extracted color as the detected color of the setting information. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The second detection means detects the specific object using a trained model that has been trained to be able to detect the specific object using a deep neural network. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The second detection means detects the specific object using depth information acquired from a captured image of the specific object. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The second detection means does not execute the second process on the second captured image when a difference between a reference color distribution, which is a color distribution of a first captured image in which the specific object is not detected, and a color distribution of a second captured image captured after the first captured image is smaller than a predetermined threshold value. 7. The information processing device according to any one of configurations 1 to 6. (method) a first detection step of repeatedly performing a first process of detecting a specific object from a real space by using setting information stored in a storage unit; a second detection step of performing a second process that is a process of detecting the specific object from the real space and that takes a longer time than a time required for the first process; an updating step of updating the setting information based on a result of the second processing; a control step of controlling so as to perform a predetermined process based on a result of the first process; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 7. [Explanation of symbols]

[0059] 100: information processing device, 103: data storage unit, 104: color region detection unit, 108: region detection unit, 109: color update unit, 110: control unit

Claims

1. a first detection means for repeatedly performing a first process for detecting a specific object from real space using setting information stored in a storage unit; a second detection means for performing a second process asynchronously and in parallel with the first process, the second process being a process for detecting the specific object from the real space with higher accuracy than the first process and taking a longer time than the time required for the first process; an update means for dynamically updating the setting information based on a result of the second process; a control means for controlling the device to perform a predetermined process based on a result of the first process performed using the latest setting information; An information processing device comprising:

2. The predetermined processing is processing for generating a display image by combining a virtual object with the captured image of the real space.

2. The information processing apparatus according to claim 1, wherein:

3. the setting information includes color information of the specific object; the first detection means detects the specific object using color information of the specific object; The updating means updates the setting information based on color information of the specific object detected in the second process.

2. The information processing apparatus according to claim 1, wherein:

4. the setting information includes information on a detection color to be detected as the color of the specific object and information on a non-detection color that is not to be detected as the color of the specific object, When the color extracted from the specific object detected in the second process is the non-detected color, the updating means does not register the extracted color as the detected color of the setting information.

2. The information processing apparatus according to claim 1, wherein:

5. The second detection means detects the specific object using a trained model that has been trained to be able to detect the specific object using a deep neural network without using color information of the specific object.

4. The information processing apparatus according to claim 3,

6. The second detection means detects the specific object using depth information acquired from a captured image of the specific object without using color information of the specific object.

4. The information processing apparatus according to claim 3,

7. The second detection means does not execute the second process on the second captured image when a difference between a reference color distribution, which is a color distribution of the first captured image in which the specific object is not detected, and a color distribution of the second captured image captured after the first captured image is smaller than a predetermined threshold.

4. The information processing apparatus according to claim 3,

8. The first detection means performs the first processing on the first captured image and also performs the first processing on the second captured image.

8. The information processing apparatus according to claim 7,

9. The second detection means detects the area of ​​the specific object using a trained model that has been trained to be able to detect the specific object when the color of the specific object changes due to a change in optical conditions, The updating means updates the setting information based on color information extracted from the region of the specific object.

2. The information processing apparatus according to claim 1, wherein:

10. a first detection step of repeatedly performing a first process of detecting a specific object from real space using setting information stored in a storage unit; a second detection step of performing a second process, which is a process for detecting the specific object from the real space with higher accuracy than the first process and takes a longer time than the time required for the first process, asynchronously and in parallel with the first process; an updating step of dynamically updating the setting information based on a result of the second processing; a control step of controlling the device to perform a predetermined process based on a result of the first process performed using the latest setting information; An information processing method comprising:

11. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 9.

Images