Image processing device, its control method, program

Abstract

To provide means which acquires a pixel value of each color patch in a color chart regardless of inclination and a rotation angle of the color chart included in an image.SOLUTION: An image processing device for acquiring a pixel value of each color patch of a color chart in an image comprises: means which inputs an image including a color chart; means which executes deduction by means of a partial region learning model to the input image; means which acquires position information of a plurality of partial regions of the color chart in the input image from the deduction execution result of the partial region learning model; means which calculates the position of each color patch from the position information of each partial region; and means which acquires a pixel value of each color patch on the basis of the position of each color patch. The partial region learning model labels the plurality of partial regions of the color chart.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This invention relates to a technique for detecting specific objects from images. [Background technology]

[0002] In video production, it is common to use multiple imaging devices for shooting. Even when shooting the same subject, the color and brightness characteristics obtained from the imaging devices will differ due to individual differences in the imaging devices and lenses used, as well as differences in ambient lighting. To match the characteristics of the images output from the imaging devices, a color matching system using a color chart exists.

[0003] In color matching systems, one method involves capturing a scene containing the same color chart with multiple cameras, generating a 3DLUT (Look Up Table) that compensates for color differences from the pixel values ​​of each color patch in the color chart, and applying it to one of the cameras. This method is sometimes called color matching.

[0004] In machine learning-based object detection, there is a method in which training data labeled with the objects to be detected within an image is input, and a learning model that has learned to detect the objects is generated. This method performs inference using the learning model on arbitrary data in which the objects to be detected are included in the image, and outputs the detection location of the objects as a bounding box.

[0005] In the object detection methods described above, when performing object detection using training data in which the entire color chart is labeled, the output of object detection (bounding box) can be directly interpreted as the precise position of the color chart, since the color chart itself is rectangular. Therefore, it is possible to identify the position of each color patch from the relative position information of the color chart and calculate the pixel value of each color patch.

[0006] However, if the color chart in the image is not aligned (rotated or tilted), the object detection method described above cannot directly interpret the output as the accurate position of the color chart, and therefore cannot determine the position of each color patch from the relative position information of the color chart.

[0007] Patent Document 1 describes a technique for acquiring information about a target object using multiple sub-regions. In Patent Document 1, the multiple sub-regions are selected so that they include both overlapping and non-overlapping regions. This allows for obtaining multiple inference results in regions where the areas of interest in the image overlap. Furthermore, inference results for the surrounding regions can also be obtained.

[0008] Patent Document 2 describes a technique for accurately detecting an object by using multiple deep learning models. In Patent Document 2, an object 1 is detected by a first trained model. Furthermore, by using a second trained model to detect an object 2 contained within the object 1 from an image containing the object 1, it becomes possible to accurately detect the object. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2021-09314 [Patent Document 2] Japanese Patent Publication No. 2020-197795 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] However, in the technology described in Patent Document 1, when selecting multiple sub-regions, it is necessary to include both overlapping and non-overlapping regions. On the other hand, since the positions of all color patches included in the color chart do not overlap, it is not possible to determine the positions of all color patches from the output results of multiple sub-regions, and therefore it is not possible to obtain the pixel values ​​of each color patch.

[0011] In addition, in the technique of Patent Document 2, since the object to be labeled by the second learned model is the entire detection object 2, the detection object 1 is a color chart and the detection object 2 is a single color patch. Only with the position information of the detection object 2, the positions of all the color patches cannot be specified, and the pixel values of each color patch cannot be obtained.

[0012] As described in the above example of the color chart, when detecting a specific object from a captured image, if the object is not imaged directly facing the imaging device, it may not be detected appropriately.

[0013] Therefore, an object of the present invention is to provide means for appropriately detecting even when the imaged object is not directly facing the imaging device.

Means for Solving the Problem

[0014] To solve the above problems, an image processing apparatus according to the present invention includes an input means for inputting an image including a color chart, and based on the result of inference by a first learning model for detecting the entire area of the color chart with respect to the input image, a first specifying means for specifying the position of the color chart in the input image, a generating means for generating a cut-out image obtained by cutting out the area including the color chart from the input image based on the position of the color chart specified by the first specifying means, a second specifying means for specifying the positions of a plurality of partial areas of the color chart in the input image based on the result of inference by a second learning model for detecting the partial area of the color chart with respect to the cut-out image, a third specifying means for specifying the positions of the plurality of color patches from the position information of the plurality of partial areas, and an obtaining means for obtaining the pixel values of the plurality of color patches based on the positions of the color patches. 3 specifying means, and an obtaining means for obtaining the pixel values of the plurality of color patches based on the positions of the color patches.

Effect of the Invention

[0015] According to the present invention, even when the imaged object is not facing the imaging device directly, a specific object can be appropriately detected.

Brief Description of the Drawings

[0016] [Figure 1] It is a conceptual diagram showing the system configuration in the first embodiment. [Figure 2] It is a block diagram showing the configuration of the communication device 200 in the first embodiment. [Figure 3] It is a software configuration diagram of the communication device 200 in the first embodiment. [Figure 4] It is a flowchart of a method for creating a learning model in the first embodiment. [Figure 5] It is a diagram showing the color chart 500 in the first embodiment. [Figure 6] It is a diagram showing a method for labeling the first learning model in the first embodiment. [Figure 7] It is a diagram showing a method for labeling the second learning model in the first embodiment [[ID=2-7]] [Figure 8] It is a diagram showing the screen UI of the color matching application 301 in the first embodiment. [Figure 9] It is a processing flow at the time of acquiring image information of the color matching application 301 in the first embodiment. [Figure 10] It is a processing flow for acquiring position information of the cut-out image and the color patch of the color matching application 301 in the first embodiment. [Figure 11] It is a processing flow of the cut-out image generation unit 304 in the first embodiment. [Figure 12] It is a processing flow of the detection position information acquisition unit 306 in the first embodiment. [Figure 13] It is an explanatory diagram of a method for generating a cut-out image in the first embodiment. [Figure 14] It is a processing flow of the color patch position calculation unit 305 in the first embodiment. <000012-8> [Figure 15] This is an explanatory diagram of the method for calculating the position of each color patch in the color chart 500 in the first embodiment. [Figure 16] This is a processing flow for displaying a preview image in the color matching application 301 in the first embodiment. [Figure 17] This is the processing flow of the pixel value acquisition unit 307 in the first embodiment. [Figure 18] This is a diagram showing the 3DLUT1801 in the first embodiment. [Figure 19] This is a software configuration diagram of the communication device 200 in the second embodiment. [Figure 20] This is the processing flow for acquiring the position information of a color patch in the second embodiment. [Figure 21] This is the processing flow of the detection position information acquisition unit 305 in the second embodiment. [Figure 22] This is a diagram showing the image information list in the first embodiment. [Figure 23] This figure shows the inference results in the first embodiment. [Figure 24] This is a diagram showing a 3DLUT in the first embodiment. [Modes for carrying out the invention]

[0017] The embodiments for carrying out the present invention will be described in detail below with reference to the attached drawings or tables. The embodiments described below are merely examples of means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. Furthermore, the embodiments can be combined as appropriate.

[0018] [First Embodiment] <System Configuration> Figure 1 is a schematic diagram illustrating the system configuration of the present invention. In Figure 1, 100 is an imaging device such as a professional video camera. The imaging device 100 is assumed to have communication capabilities. 200 is a communication device acting as a mobile terminal, which is an image processing device equipped with communication capabilities, such as a smartphone or computer. 400 is a network, which is a communication line for connecting devices equipped with communication capabilities.

[0019] In Figure 1, the imaging device 100 and the communication device 200 are connected wirelessly via a network 400 using a connection method such as Wi-Fi. The imaging device 100 can transfer image files to the communication device 200.

[0020] <Hardware Configuration> Figure 2 is a block diagram showing an example of the hardware configuration of the communication device 200 in this embodiment. While a mobile terminal is described here as an example of the communication device 200, the communication device 200 is not limited to this. For example, the communication device 200 may be a tablet device or a personal computer.

[0021] The control unit 201 controls each part of the communication device 200 according to the input signals and the program described later. Alternatively, instead of the control unit 201 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0022] The wireless communication unit 202 is an interface for communicating with external devices. The communication device 200 in this embodiment can exchange data with the imaging device 100 via the wireless communication unit 202.

[0023] In this embodiment, the wireless communication unit 202 is an antenna, and the communication device 200 can be connected to the imaging device 100 via the antenna. The connection between the imaging device 100 and the communication device 200 may be direct or via an access point. For data communication, for example, PTP / IP (Picture Transfer Protocol over Internet Protocol) via a wireless LAN can be used. The wireless communication unit 202 also includes a USB interface via a USB (Universal Serial Bus) cable for connection to external devices. However, communication with external devices is not limited to this. For example, the wireless communication unit 202 may include wireless communication modules such as an infrared communication module, a Bluetooth® communication module, or WirelessUSB.

[0024] The display unit 203 displays image data, text for interactive operation, etc. Note that the display unit 203 does not necessarily need to be part of the communication device 200. The communication device 200 can connect to the display unit 203 and only needs to have at least a display control function to control the display of the display unit 203.

[0025] The main memory 204 is used as image display memory for the display unit 203, a work area for the control unit 201, and so on.

[0026] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 205 stores the operating system (OS), which is the basic software executed by the control unit 201, and applications that work in conjunction with the OS to realize advanced functions.

[0027] In this embodiment, the non-volatile memory 205 stores a color matching application 301 (hereinafter referred to as the color matching application 301) that generates 3DLUT data for color matching using two images from the image data acquired from the imaging device 100.

[0028] Furthermore, a file management application 310 (hereinafter referred to as the file management application 310) for managing image files output from the imaging device 100 and a machine learning library 312 are stored.

[0029] The control unit 206 is used to receive instructions from the user for the communication device 200. The control unit 206 includes, for example, a power button for the user to instruct the communication device 200 to turn on or off, and an operating component such as a touch panel formed on the display unit 203.

[0030] The recording unit 207 can record image data acquired from the imaging device 100. The recording unit 207 may be configured to be detachable from the communication device 200, or it may be built into the communication device 200. In other words, the communication device 200 only needs to have means to access the recording unit 207.

[0031] The public network connection unit 210 is an interface used when conducting public wireless communication. The communication device 200 can communicate with other devices via the public network connection unit 210. In this case, the control unit 201 enables communication by inputting and outputting audio signals via the speaker 208 and microphone 209.

[0032] The above is a description of the communication device 200.

[0033] <Software Configuration> Next, the software configuration diagram of the communication device 200 in this embodiment will be described. For convenience, the following description may include statements that suggest the software is primarily responsible for processing; however, the actual processing is achieved by the control unit 201 controlling each part of the communication device 200 according to the program and input signals from each part. Furthermore, while the description may include statements that suggest the user is primarily responsible for processing, this means that the control unit 201 receives instructions from the user via the operation unit 206 and performs various processing actions.

[0034] Figure 3 is a software configuration diagram of the communication device 200. Various fixed data and firmware are stored in the non-volatile memory 205 of the communication device 200.

[0035] The recording unit 207 stores the OS 316, which is the basic software executed by the control unit 201, and the file management application 310 and machine learning library 312, which are application software executed by the control unit 201. In addition, the color matching application 301 is stored as application software, which generates 3DLUT data for color matching using two images from the image data acquired from the imaging device 100.

[0036] When the user turns on the power switch included in the control unit 206 of the communication device 200, the communication device 200 starts up, and the control unit 201 reads the OS 316 from the recording unit 207, loads it into the main memory 204, and executes it. It also controls various parts of the communication device 200 according to the OS 316 and the applications installed on the OS.

[0037] In the following diagrams, the operation of the control unit 201 according to an application (or application function, or OS or OS service, etc.) may be described as "the application (or application function, or OS or OS service, etc.) performs the predetermined processing." The OS 316 not only has the function of controlling each part of the communication device 200, but also provides various services to the application.

[0038] The file management application 310 has a file management unit 311 that manages a list of files recorded in the recording unit 207. It also controls the reading of files managed by the file management function from the recording unit 207 and the writing of files to the recording unit 207 in response to requests from the application.

[0039] The machine learning library 312 includes an inference execution unit 313, a first learning model storage unit 314, and a second learning model storage unit 315 for detecting the positional information of objects to be detected in an image using a neural network. A neural network is a network model that mimics the neural network in the human brain. When the inference execution unit 313 receives an inference execution instruction and a path to the image data to be inferred from an external source, it retrieves a learning model from either the first learning model storage unit 314 or the second learning model storage unit 315 and performs inference on the received image data.

[0040] Inference refers to executing a machine learning program based on a pre-generated learning model for detecting specific objects. In this embodiment, the machine learning program is a program that outputs location information of the object to be detected in the aforementioned learning model, and it is possible to detect the location information of the object to be detected by inputting image data.

[0041] In this embodiment, two learning models can be stored, but it is also possible to store three or more.

[0042] Next, we will describe the color matching application 301, which is a feature of this embodiment.

[0043] The color matching application 301 generates a screen to be displayed on the display unit 203 in the display control unit 302. The configuration of the generated screen will be described later in the explanation of Figure 8.

[0044] In response to user input, the display control unit 302 sends a request to the file management unit 311 of the file management application 310 to obtain image file list information for the specified folder, and retrieves the file list in order to display the image file list for the specified folder.

[0045] Furthermore, the display control unit 302 sends a request to the thumbnail image generation unit 303 to generate a thumbnail image for the specified image file in order to display the thumbnail list image, and obtains the thumbnail image.

[0046] Furthermore, the display control unit 302 sends a request to the cropped image generation unit 304 to generate a cropped image in order to display the cropped image, and acquires the cropped image.

[0047] Furthermore, the display control unit 302 sends a request to the color patch position calculation unit 305 to acquire color patch position information in order to display the position information of the color patch, and acquires the position information of the color patch.

[0048] Furthermore, the display control unit 302 sends a request to the pixel value acquisition unit 307 to acquire the pixel value for each color patch and receives a notification that the acquisition of pixel value information is complete.

[0049] Furthermore, the display control unit 302 sends a 3DLUT generation request to the 3DLUT generation unit 308 and receives a notification that 3DLUT generation is complete.

[0050] Furthermore, the display control unit 302 sends a preview target image generation request to the preview target image generation unit 309 in order to display the preview target image after 3DLUT application, and acquires the preview target image.

[0051] When the thumbnail image generation unit 303 receives a request from the display control unit 302 to generate a thumbnail image of a specified image file, it generates a thumbnail image by referring to the image information of the specified image file in the file management application 310 and transmits it to the display control unit 302.

[0052] When the cropped image generation unit 304 receives a cropped image generation request and a specified image path from the display control unit 302, it sends a color chart position information acquisition request to the detection position information acquisition unit 306 and receives color chart position information from the detection position information acquisition unit 306.

[0053] The cropped image generation unit 304 generates a cropped image based on the color chart position information received from the detection position information acquisition unit 306 and transmits the cropped image to the display control unit 302.

[0054] Furthermore, the cropped image generation unit 304 transmits the generated cropped image data to the file management unit 311.

[0055] The method for generating the cropped images will be described later in Figure 13.

[0056] When the color patch position calculation unit 305 receives a request to acquire color patch position information and a specified cropped image path from the display control unit 302, it sends a request to the detection position information acquisition unit 306 to acquire position information for multiple partial regions within the color chart 500.

[0057] When the color patch position calculation unit 305 receives position information for multiple sub-regions within the color chart 500 from the detection position information acquisition unit 306, it calculates the position of each color patch within the color chart and transmits the color patch position information to the display control unit 302.

[0058] The method for calculating the position of each color patch within the 500-color chart will be described later in Figure 15.

[0059] When the detection position information acquisition unit 306 receives a request to acquire position information of a detected object from the cropped image generation unit 304 or the color patch position calculation unit 305, it sends an inference execution instruction for the specified learning model to the inference execution unit 313 of the machine learning library 312.

[0060] The processing flow of the detection position information acquisition unit 306 will be described later in Figure 12.

[0061] When the pixel value acquisition unit 307 receives a request from the display control unit 302 to acquire the pixel values ​​of each color patch and a specified cropped image path, it acquires the pixel values ​​of each color patch from the specified cropped image path and transmits the pixel value information of each color patch to the 3DLUT generation unit 308. The pixel value acquisition unit 307 also sends a notification to the display control unit 302 that the acquisition of pixel value information is complete.

[0062] The method for obtaining the pixel values ​​of each color patch will be described later in Figure 17.

[0063] When the 3DLUT generation unit 308 receives a 3DLUT generation request and pixel value information for each color patch of the specified image from the display control unit 302, it generates a 3DLUT based on the pixel value information for each color patch of the specified image and transmits the 3DLUT data to the preview target image generation unit 309.

[0064] Furthermore, the 3DLUT generation unit 308 sends a 3DLUT generation completion notification to the display control unit 302 and also sends the 3DLUT data to the main memory 204 for temporary storage of the LUT data.

[0065] The method for generating a 3DLUT will be described later in Figure 18.

[0066] When the preview target image generation unit 309 receives a preview target image generation request and a path for the image (target image) to be color-matched from the display control unit 302, it sends a 3DLUT data acquisition request to the 3DLUT generation unit 308 and acquires 3DLUT data from the 3DLUT generation unit 308.

[0067] A preview target image is generated from the target image path and 3DLUT data and transmitted to the display control unit 302.

[0068] The processing flow for displaying the preview target image will be described later in Figure 16.

[0069] <How to create a learning model> Next, using Figure 4, we will explain the flowchart for creating learning models to generate the first learning model 314 and the second learning model 315.

[0070] In this embodiment, the learning model is a machine learning model that has learned the position of the object to be detected using a neural network, and the position of the object to be detected is detected by performing inference using the learning model.

[0071] In the learning model generation tool, users can specify the location of the object to be detected using a bounding box, and it is also possible to train a single learning model to detect the locations of multiple objects.

[0072] The flowchart shown in Figure 4 outlines the procedure for creating a learning model for object detection by labeling images containing objects with the location of the objects as training data.

[0073] In this embodiment, the process shown in Figure 4 is performed using a learning model generation tool. The learning model generation tool is not located on the communication device 201, but is prepared as a PC application or cloud service installed on a separate computer. The inference execution unit 313 then reads the learning model generated by the learning model generation tool and can perform the corresponding inference. Of course, the communication device may also be configured to have the learning model generation tool.

[0074] In S401 of Figure 4, the user operates a learning model generation tool controlled by a computer processor and inputs an image containing the object to be detected. The learning model generation tool accepts the image input. In this embodiment, the object to be detected is the entire color chart or a part of it. This will be explained later.

[0075] In S402 of Figure 4, the learning model generation tool displays the input image and accepts the labeling of the location of the object to be detected. The user operates the computer and labels the location of the object to be detected contained in the image that was input in S401. The specific methods of labeling when generating the first and second learning models will be described later in Figures 6 and 7.

[0076] The processes described in S401 to S402 above are repeated for all images to be created as a dataset. A dataset is a collection of image data in which the positions of the objects to be detected are labeled, and it is used when creating a learning model.

[0077] In S403 of Figure 4, the user inputs a dataset containing labeled locations of objects to be detected into the learning model generation tool and instructs it to generate a learning model. The learning model generation tool, having received the input and instructions, creates a learning model that outputs the locations of objects to be detected within the image.

[0078] <Color Chart> Next, the color chart 500 in this embodiment will be described using Figure 5.

[0079] The color chart 500 shown in Figure 5 has color patches 501 to 524 arranged in a straight line.

[0080] As an example of the color chart in this embodiment, we describe a color chart consisting of a total of 24 color patches, with 6 color patches in each row and 4 color patches in each column, but the color chart is not limited to this. For example, a color chart only needs to have at least M × N (where M and N are integers of 2 or more) color patches arranged in each row and each column, and the total number of color patches in each row and each column of the color chart may be more or less than 24.

[0081] <Labeling method for the first learning model> Next, we will explain the labeling method for the first learning model using Figure 6.

[0082] The labeling range 601 shown in Figure 6 represents the labeling range used when creating the dataset to generate the first learning model.

[0083] The first learning model in this embodiment is a learning model for detecting the entire color chart 500 contained within an image.

[0084] When creating the first learning model in S403, input an image containing color chart 500 in S401.

[0085] The first learning model is a rectangle-based object detection model, and in S402, labeling is performed so that the labeling range 601 coincides with the four corners of the color chart 500.

[0086] Using the method described above, the user inputs the labeled dataset into the learning model generation tool and instructs S403 to generate a learning model capable of detecting the entire color chart 500 contained within the input image.

[0087] <Labeling method for the second learning model> Next, we will explain the labeling method for the second learning model using Figure 7. The labeling ranges 701 to 704 shown in Figure 7 represent the range of each label when creating the dataset used to generate the second learning model.

[0088] The second learning model in this embodiment is a learning model for detecting multiple subregions from the cropped image of the color chart 500 obtained by the cropped image generation unit 304.

[0089] When creating a second learning model in S403, input the cropped image of color chart 500 in S401.

[0090] The second learning model is a rectangle-based object detection model, which labels multiple sub-regions of the color chart 500 in S402.

[0091] In S402, labeling is performed so that the labeling area 701 coincides with the four corners of color patch 501 on color chart 500.

[0092] Furthermore, in S402, labeling is performed so that the labeling area 702 coincides with the four corners of the color patch 506 of the color chart 500.

[0093] Furthermore, in S402, labeling is performed so that the labeling area 703 coincides with the four corners of the color patch 519 of the color chart 500.

[0094] Furthermore, in S402, labeling is performed so that the labeling area 704 coincides with the four corners of the color patch 524 of the color chart 500.

[0095] Using the method described above, the user inputs the labeled dataset into the learning model generation tool and instructs S403 to generate a learning model capable of detecting multiple subregions within the extracted images of the color chart 500.

[0096] In this embodiment, an example of the labeling ranges 701 to 704 is described as labeling individual color patches at the four corners of the color chart 500, but the range and position of each labeling are not limited to this. For example, each labeling range may not be a single color patch, but rather multiple color patches such as a 2x2 grid. In other words, the labeling ranges for the top left corners are color patches 501, 502, 507, and 508; the top right corners are color patches 505, 506, 511, and 512; the bottom left corners are color patches 513, 514, 519, and 520; and the bottom right corners are color patches 517, 518, 523, and 524.

[0097] <Color Matching App Screen> Next, the configuration of the color matching application 301 will be described using Figure 8. Figure 8 is a schematic diagram showing the display state and functions of the operation screen of the color matching application 301.

[0098] In Figure 8(a), 801 is the screen displayed after launching the color matching app 301, and consists of a folder selection button 802 and a thumbnail image display area 803. By tapping the folder selection button 802, a folder saved in the file management unit 311 of the file management app 310 can be selected. The processing flow related to folder selection will be described later in the explanation of Figure 9.

[0099] The color matching application 301 generates thumbnail images using the thumbnail image generation unit 303 based on the information of the images in the selected folder, and displays the thumbnail images of the images in the folder on 803.

[0100] In Figure 8(b), 804 is the UI displayed when the thumbnail image of the image used as the reference for color matching (hereinafter referred to as the reference image) is tapped, and 805 is the UI displayed when the thumbnail image of the image to be color-matched (hereinafter referred to as the target image) is tapped. The reference image 807 and target image 808 selected by tapping are displayed in the selected thumbnail image display section 806.

[0101] Furthermore, 809 is an image showing the color chart of the reference image 807 extracted by the extracted image generation unit 304 and the color patch position information output by the color patch position calculation unit 305.

[0102] 810 is an image showing the color chart of the target image 808 extracted by the extracted image generation unit 304 and the color patch position information output by the color patch position calculation unit 305.

[0103] 811 is a preview button. When the preview button 811 is tapped, a 3DLUT is generated from the two selected images, the reference image 807 and the target image 808, and the preview image display section 812 shown in Figure 8(c) is displayed.

[0104] In Figure 8(c), 812 is the preview image display area, and when the preview image display button 811 is tapped, the reference image 807 and the target image to which the 3DLUT has been applied are displayed.

[0105] 813 is the 3DLUT save button, and 814 is the 3DLUT save confirmation dialog. When you tap the 3DLUT save button 813, the 3DLUT save confirmation dialog 814 will be displayed superimposed on screen 801.

[0106] 815 is the cancel button for 3DLUT saving; tapping the 815 cancel button will close the dialog box.

[0107] 816 is a 3DLUT save button that downloads the generated 3DLUT to the file management app. Tapping the 3DLUT save button 816 saves the 3DLUT to the file management section 311 of the file management app 310.

[0108] <Processing to retrieve image information from a specified folder> Next, using Figure 9, we will explain the processing flow for selecting a folder from the folder list recorded in the file management unit 311 of the file management application 310 and obtaining image information within the specified folder.

[0109] The processing flow shown in Figure 9 is a process that is executed in the color matching application 301 when the operation unit 206 detects a tap on the folder selection button 802 while the screen 801 shown in Figure 8(a) is displayed.

[0110] In S901 of Figure 9, the color matching application 301 requests the file management unit 311 to retrieve file list information from the folders recorded in the file management unit 311 in order to obtain file list information within a specified folder.

[0111] In S902 of Figure 9, the color matching application 301 obtains a list of files in the specified folder from the file management unit 311.

[0112] Once the file list information is obtained, the processes shown in S903 to S904 are executed for all files included in the file list information.

[0113] In step S903 of Figure 9, the color matching application 301 refers to Figure 22 to determine whether the file is in an image format. If the file is in an image format, it obtains image information from the file management unit 311 (S904). If the file is not in an image format, it terminates processing the file. Whether the file is in an image format is determined by whether the file extension is a predetermined extension.

[0114] By performing the above S903-S904 processes on all files in the specified folder, the color matching application 301 obtains image information from the file management unit 311 within the specified folder.

[0115] Through the above process, the color matching application 301 retrieves image information from the specified folder, which is recorded in the file management unit 311 of the file management application 310.

[0116] Next, using Figure 22, we show an example of an image information list that the color matching application 301 obtains from the file management unit 311 of the file management application 310.

[0117] Figure 22 shows that each file stored in the file management unit 311 includes image ID, image path, and thumbnail image data information.

[0118] When the color matching application 301 exchanges image data with the file management unit 311, it refers to the image ID and thumbnail image data associated with the image path from the image information list received from the file management unit 311.

[0119] <Processing to acquire position information of cropped images and color patches> Next, the process for displaying the selected thumbnail image display section 806 shown in Figure 8(b) in the color matching application 301 will be explained using Figures 10 to 12.

[0120] Figure 10 shows the processing flow for obtaining the position information 809 of the cropped image and color patch of the reference image 807, and the position information 810 of the cropped image and color patch of the target image 808, which are to be displayed in the selected thumbnail image display section 806.

[0121] The processing flow shown in Figure 10 is a process that is executed in the color matching application 301 when the operation unit 206 detects a tap on the reference selection image 804 and the target selection image 805 while the thumbnail image display section 803 in Figure 8(b) is being displayed.

[0122] In S1001 of Figure 10, the color matching application 301 requests the cropped image generation unit 304 to acquire a cropped image of the color chart 500, and inputs the paths of the reference selection image 804 and the target selection image 805.

[0123] In step S1002 of Figure 10, the color matching application 301 obtains a cropped image of the color chart 500 from the cropped image generation unit 304. The processing flow of the cropped image generation unit 304 will be described later in Figure 11.

[0124] In step S1003 of Figure 10, the color matching application 301 requests the color patch position calculation unit 305 to acquire color patch position information and inputs the paths of the cropped image of the reference selection image 804 and the cropped image of the target selection image 805.

[0125] In S1004 of Figure 10, the color matching application 301 acquires the color patch position information of the color chart 500 calculated by the color patch position calculation unit 305. The method for calculating the color patch position information will be described later in Figure 15.

[0126] In step S1005 of Figure 10, the color matching application 301 displays the cropped image and color patch position information of the color chart 500 acquired in steps S1002 and S1004 in the selected thumbnail image display section 806.

[0127] Through the above process, the color matching application 301 obtains the position information of the cropped image and color patches.

[0128] <Processing by the cropped image generation unit 304> Next, using Figure 11, we will explain the processing flow when the cropped image generation unit 304 receives a request to acquire a cropped image from the color matching application 301.

[0129] The processing flow shown in Figure 11 is a process that is executed in the color matching application 301 when the cropped image generation unit 304 receives a request from the display control unit 302 to acquire a cropped image.

[0130] In S1101 of Figure 11, the cropped image generation unit 304 requests the detection position information acquisition unit 306 to acquire position information of the color chart 500 in order to generate a cropped image of the color chart 500 contained within the image. At this time, the detection target is specified as the entire color chart 500. The processing performed by the detection position information acquisition unit 306 upon receiving the request will be described later in Figure 12.

[0131] In step S1102 of Figure 11, the cropped image generation unit 304 acquires the position information of the color chart 500 contained in the image from the detected position information acquisition unit 306.

[0132] In step S1103 of Figure 11, the cropped image generation unit 304 requests the file management unit 311 to obtain the input image path based on the input image path received in response to the cropped image acquisition request from the color matching application 301, and acquires the specified image from the file management unit 311.

[0133] In S1104 of Figure 11, the cropped image generation unit 304 generates a cropped image based on the position information of the color chart 500 for the specified image. The method of generating the cropped image in the cropped image generation unit 304 will be described later in Figure 13.

[0134] In step S1105 of Figure 11, the cropped image generation unit 304 saves the generated cropped image to the file management unit 311.

[0135] In S1106 of Figure 11, the cropped image generation unit 304 stores the path of the generated cropped image, associating it with the image ID in the image information list in Figure 22.

[0136] Through the above process, the cropped image generation unit generates cropped images of the color chart 500 contained in the image.

[0137] <Processing flow of the detection position information acquisition unit 306> Next, using Figure 12, the processing flow when the detection position information acquisition unit 306 receives a request to acquire position information of the color chart 500 from the cropped image generation unit 304 or a request to acquire position information of multiple partial regions of the color chart 500 from the color patch position calculation unit 305 will be explained.

[0138] The processing flow shown in Figure 12 is a process that is executed when the detection position information acquisition unit 306 receives a request to acquire detection position information from the cropped image generation unit 304 or the color patch position calculation unit 305.

[0139] In S1201 of Figure 12, the detection position information acquisition unit 306 determines whether the detected position target of the received acquisition request is the entire color chart 500. If the detected position target is the entire color chart 500, it designates the model to be applied as the first learning model (S1202). If the detected position target is not the entire color chart 500, it designates the model to be applied as the second learning model (S1203).

[0140] In S1204 of Figure 12, the detection position information acquisition unit 306 instructs the inference execution unit 313 of the machine learning library 312 to perform inference using the learning model specified in S1202 or S1203 in order to acquire detection position information of the specified learning model.

[0141] In step S1205 of Figure 12, the detection position information acquisition unit 306 acquires the position information of the specified learning model from the inference execution unit 313.

[0142] Through the above process, the detection position information acquisition unit 306 acquires the position information of the entire color chart 500 or the position information of multiple partial regions of the color chart 500.

[0143] <Inference execution results using the first learning model> Next, using Figure 23(a), we will explain the output parameters that can be obtained from the inference execution results using the first learning model. The inference results using the first learning model, which can detect the entire color chart 500, output five parameters: label name, starting coordinate x, starting coordinate y, width w, and height h. The starting point is the top left of the rectangle, with the right direction being the x-direction and the downward direction being the y-direction.

[0144] The label names in Figure 23(a) represent the label names for the labeling range 601 in Figure 7, and the label name is Color Chart.

[0145] In Figure 23(a), the starting coordinates x and y of the inference result represent the coordinates of the top left corner of the labeled range 601.

[0146] In Figure 23(a), the width w and height h of the inference results represent the horizontal and vertical lengths of the labeling range 601, respectively, starting from the top-left coordinate. In the inference results of Figure 23(a), the coordinates of the four corners of the labeling range 601 can be determined from the top-left coordinates x, y, width w, and height h.

[0147] Next, using Figure 23(b), we will explain the output parameters that can be obtained from the inference results using the second learning model. The inference results using the second learning model, which can detect multiple subregions within the color chart 500, output five parameters: label name, starting coordinate x, starting coordinate y, width w, and height h.

[0148] In Figure 23(b), the label names of the inference results indicate the label names for the labeling range 701 to 704.

[0149] In Figure 23(b), the label name for the labeled area 701 is Left Up (the upper left corner of the four corners).

[0150] In Figure 23(b), the label name for the labeling range 702 is Left Bottom (the bottom left of the four corners).

[0151] In Figure 23(b), the label name for the labeled area 703 is "Right Up" (the upper right corner of the four corners).

[0152] In Figure 23(b), the label name for the labeled area 704 is Right Bottom (the bottom right corner of the four corners).

[0153] In Figure 23(b), the starting coordinates x and y of the inference result represent the top-left coordinates of the labeling range 701-704.

[0154] In Figure 23(b), the width w and height h of the inference results represent the horizontal and vertical lengths of the labeling range 701-704, respectively, starting from the top-left coordinate. In the inference results of Figure 23(b), the coordinates of the four corners of the labeling range 701-704 can be determined from the top-left coordinates x, y, width w, and height h.

[0155] <Diagram explaining the method for generating cropped images> Next, using Figure 13, we will explain how to generate the cropped image of the color chart 500 generated by the cropped image generation unit 304.

[0156] Figure 13(a) shows an example of an image containing the color chart 500.

[0157] 1301 shows the detection position result of the color chart 500 acquired by the cropped image generation unit 304 from the detection position information acquisition unit 306. As mentioned above, the cropped image generation unit 304 outputs a rectangular area as the detection position result.

[0158] In this embodiment, the detected position result 1301 overlaps with a portion of the color chart 500, but the detected position result 1301 is not limited to this. For example, if the color chart 500 included in the image is directly facing the image, the detected position result 1301 will coincide with the color chart 500. Alternatively, the detected position result 1301 may detect an area outside the color chart 500.

[0159] In S1105, the cropped image generation unit 304 saves the cropped image, which was generated based on the detected position result 1301, to the file management unit 311.

[0160] Figure 13(b) shows the cropped image of the color chart 500 generated by the cropped image generation unit 304, as well as the background area surrounding the color chart 500.

[0161] 1302 is a rectangle that includes the detected position result 1301 and the background area outside of the detected position result 1301, and provides a so-called margin around the detected position result 1301. The cropped image generation unit 304 can generate a cropped image that includes the color chart 500 and the pixels outside of it by cropping out the rectangle 1302.

[0162] <Processing by the color patch position calculation unit 305> Next, using Figure 14, we will explain the processing flow when the color patch position calculation unit 305 receives a request to acquire color patch position information from the color matching application 301.

[0163] The processing flow shown in Figure 14 is a process that is executed in the color matching application 301 when the color patch position calculation unit 305 receives a request to acquire the color patch position from the display control unit 302.

[0164] In S1401 of Figure 14, the color patch position calculation unit 305 requests the detection position information acquisition unit 306 to acquire position information for multiple subregions within the color chart 500 in order to obtain position information for multiple subregions within the color chart 500. At this time, the detection target is specified as multiple subregions within the color chart 500. The processing of the detection position information acquisition unit 306 upon receiving the request is explained in Figure 12.

[0165] In step S1402 of Figure 14, the color patch position calculation unit 305 acquires position information for multiple subregions within the color chart 500 from the detected position information acquisition unit 306.

[0166] In S1403 of Figure 14, the color patch position calculation unit 305 calculates the coordinates of the center position of each sub-region from the position information of each sub-region acquired in S1402. The method for calculating the coordinates of the center position of each sub-region from the position information of each sub-region will be described later in Figure 15(a).

[0167] In S1404 of Figure 14, the color patch position calculation unit 305 calculates the position of each color patch from the coordinates of the central position of each sub-region calculated in S1403. The method for calculating the position of each color patch from the coordinates of the central position of each sub-region will be described later in Figure 15(b).

[0168] Through the above process, the color patch position calculation unit 305 calculates the position of each color patch on the color chart 500.

[0169] <How to calculate the position of each color patch in the 500-color chart> Next, using Figure 15, the method for calculating the position of each color patch in the color chart 500 in the color patch position calculation unit 305 will be explained.

[0170] Figure 15(a) shows an example of the position information of multiple subregions within the color chart 500 obtained by the color patch position calculation unit 305 from the detected position information acquisition unit 306. The position information of each subregion is obtained from the output result of inference performed by the inference execution unit 313 using a second learning model.

[0171] Numbers 1501 to 1504 show the detection position results for each sub-region obtained from the detection position information acquisition unit 306.

[0172] The detected position result 1501 includes the entire color patch 501 of the color chart 500 within the rectangle of the detected position result.

[0173] Furthermore, the detected position result 1502 includes the entire color patch 506 of the color chart 500 within the rectangle of the detected position result.

[0174] Furthermore, the detected position result 1503 includes the entire color patch 519 of the color chart 500 within the rectangle of the detected position result.

[0175] Furthermore, the detected position result 1504 includes the entire color patch 524 of the color chart 500 within the rectangle of the detected position result.

[0176] Points A to D in Figure 15(a) represent the coordinates of the centers of the rectangles generated based on the detected position results 1501 to 1504.

[0177] Figure 15(b) shows a single color patch 1505 and its detected position result 1506 as an example to illustrate how to determine the center coordinates of a rectangle from the detected position result rectangle.

[0178] Assuming that the top-left coordinate of the detected position result 1506 is P(i, j) and that it is a square with side lengths d, in S1403, the color patch position calculation unit 305 can calculate the center coordinate O of the detected position result 1506 using the following equation (1).

[0179] By using equation (1), it is possible to determine the center coordinates of each of the detected position results 1501 to 1504 in Figure 15(a), and the color patch position calculation unit 305 can obtain the center coordinates of each rectangle of each detected position result. O(x, y)=P(i+d / 2, j+d / 2) (Formula 1) Figure 15(c) is a diagram shown in S1504 to explain how to determine the center coordinates of each color patch from the distance information of the center coordinates of the detected position results 1501 to 1504 in Figure 15(b).

[0180] Here, as an example, the center coordinates (x) of an arbitrary color patch. mn , y mn The following explains how to find the value of the color patch. The subscripts m and n indicate the position of the color patch in the mth row and nth column, with the center coordinates of the upper left color patch 501 as the starting point and the position of the color patch being in the 1st row and 1st column.

[0181] The center coordinates (x) of any color patch mn , y mnTo obtain [[ID=]], a quadrilateral is created by connecting the center coordinates of the color patches at the four corners within the color chart 500. This quadrilateral passes through the center coordinates of each outer color patch within the color chart 500 and divides each line segment on the horizontal and vertical line segments into equal parts at the center coordinates of each color patch.

[0182] Therefore, since there are 6 center coordinates on the horizontal line segment, when this line segment is divided into equal parts by 6 center coordinates, the number of equal parts in the horizontal line segment is 5. Similarly, since there are 4 center coordinates on the vertical line segment, when this line segment is divided into equal parts by 4 center coordinates, the number of equal parts in the vertical direction is 3. That is, when a line segment is divided into equal parts by N center coordinates, the number of equal parts is N - 1.

[0183] First, to obtain the center coordinates (x mn , y mn ) of an arbitrary color patch, from the center coordinates (x 11 , y 11 ) of the color patch 501 and the center coordinates (x 14 , y 14 ) of the color patch 519, the center x - coordinate and center y - coordinate of the color patches 507 and 513 are obtained using the following equations (2) and (3).

[0184]

Number

[0185] Furthermore, from the center coordinates (x 61 , y 61 ) of the color patch 506 and the center coordinates (x 64 , y 64 ) of the color patch 524, the center x - coordinate and center y - coordinate of the color patches 512 and 518 are obtained using the following equations (4) and (5).

[0186]

Number

[0187] By using equations (2) to (5) above, the center coordinates of color patches 507, 512, 513, and 518, which lie on the vertical line segments of the rectangle formed by connecting the center coordinates of each of the four corner color patches within color chart 500, can be determined.

[0188] Next, the center coordinates (x) of the color patch in the nth column. 1n , y 1n ), (x 6n , y 6n From the horizontal line segment connecting ) to the center coordinates (x) of any color patch mn , y mn To find ), we use the following equations (6) and (7).

[0189]

number

[0190] By using equations (2) to (7) described above, the center coordinates of all color patches in the 500-color chart can be determined.

[0191] For example, the center coordinates (x) of the color patch 508 shown in Figure 15(c) 32 , y 32 To find the center coordinate (x), first substitute n=2 into equations (2) to (5), and from the vertical line segment of the rectangle formed by connecting the center coordinates of each of the four corner color patches detected, 12 , y 12 ), (x 62 , y 62 )

[0192] By connecting these two points, we can identify the horizontal line segment where the center coordinates of color chart 508 are located, and on this line segment, the center coordinates (x m2 , y m2 There exists a case where (1 ≤ m ≤ 6).

[0193] Next, by substituting m=3 and n=2 into equations (6) and (7), we can obtain the center coordinates (x) of color patch 508. 32 , y 32The position of ) can be calculated.

[0194] Based on the above calculations, the color patch position calculation unit 305 can determine the center coordinates of each color patch within the color chart 500.

[0195] In the description of Figure 15 of this embodiment, the calculation method described assumes that the distance between each color patch in the color chart 500 is uniform. However, if the color chart 500 is tilted in the depth direction and appears trapezoidal or irregular in shape, a coefficient corresponding to the ratio of the lengths of each side of the rectangle formed by connecting the center coordinates of each of the detected corner color patches may be calculated, and the distance between each color patch may be weighted to determine the center coordinate of each color patch. Alternatively, the color chart 500 may be transformed to face the center using a projection transformation, and then the center coordinate of each color patch may be determined using equations (2) to (9) described above.

[0196] <Preview image display> Next, the process for displaying the preview image display section 809 shown in Figure 8(c) in the color matching application 301 will be explained using Figures 16 and 17.

[0197] The processing flow shown in Figure 16 is a process that is executed in the color matching application 301 when the operation unit 206 detects a tap on the preview image display button 808, which is displayed as shown in Figure 8(b).

[0198] In step S1601 of Figure 16, the color matching application 301 requests the pixel value acquisition unit 307 to acquire the pixel values ​​of each color patch of the color chart 500 in order to obtain the pixel values ​​of each color patch of the color chart 500.

[0199] In step S1602 of Figure 16, the color matching application 301 acquires the average pixel value information for each color patch of the color chart 500 from the pixel value acquisition unit 307. The processing flow for acquiring the average pixel value for each color patch of the color chart 500 in the pixel value acquisition unit 307 will be described later in Figure 17.

[0200] In step S1603 of Figure 16, the color matching application 301 instructs the 3DLUT generation unit 308 to generate a 3DLUT in order to obtain a 3DLUT.

[0201] In step S1604 of Figure 16, the color matching application 301 receives a notification from the 3DLUT generation unit 308 that 3DLUT generation is complete. The 3DLUT generation unit 308 also saves the generated 3DLUT data to the main memory 204.

[0202] In S1605 of Figure 16, the color matching application 301 instructs the preview image display unit 809 to generate a preview target image in order to display the reference image and the target image to which the 3DLUT has been applied (hereinafter referred to as the preview target image).

[0203] When the preview target image generation unit 309 receives a request for preview target image generation from the color matching application 301, it acquires 3DLUT data from the main memory 204.

[0204] In S1606 of Figure 16, the color matching application 301 acquires the preview target image generated by the preview target image generation unit 309. The reference image and the acquired preview target image are displayed in the preview image display section 809 by the display control unit 302.

[0205] Through the above process, the color matching application 301 displays the reference image and the preview target image in the preview image display section 809.

[0206] <Processing by the pixel value acquisition unit 307> Next, using Figure 17, we will explain the processing flow when the pixel value acquisition unit 307 receives a pixel value acquisition request from the color matching application 301.

[0207] The processing flow shown in Figure 17 is a process that is executed in the color matching application 301 when the operation unit 206 detects a tap on the preview image display button 808, which is displayed as shown in Figure 8(b).

[0208] In S1701 of Figure 17, the pixel value acquisition unit 307 acquires the central position of each color patch from the pixel value acquisition request and color patch position information received from the color matching application 301.

[0209] In step S1702 of Figure 17, the pixel value acquisition unit 307 calculates the average pixel value of the surrounding pixels at the center coordinates of each color patch. The average pixel value uses RGB values.

[0210] In step S1703 of Figure 17, the pixel value acquisition unit 307 stores the average pixel value of each color patch calculated in step S1702.

[0211] By performing the above processing steps S1701 to S1703 on the extracted images of the color chart 500 of the reference image and the target image, the average pixel value of each color patch within each extracted image can be obtained.

[0212] In S1704 of Figure 17, the pixel value acquisition unit 307 outputs the average pixel value information of each color patch in each cropped image to the 3DLUT generation unit 308.

[0213] Through the above process, the pixel value acquisition unit 307 acquires the average pixel value of each color patch in each cropped image, sends a pixel value acquisition completion notification to the display control unit 302, and outputs it to the 3DLUT generation unit 308.

[0214] <3DLUT data> Next, using Figure 24, we will explain an example of 3DLUT data that the 3DLUT generation unit 308 generates when it receives a 3DLUT generation request from the color matching application 301.

[0215] The 3DLUT data shown in Figure 24 is generated by the 3DLUT generation unit 308, and is generated from the RGB values ​​of 24 color patches in the color chart 500 of the reference image and target image acquired from the pixel value acquisition unit 307. The method for calculating each RGB value as 3DLUT data will be described later in Figure 18.

[0216] 3DLUT can correct the color of a target image to match a reference image by taking the RGB values ​​of a target image as input and converting them to the corresponding RGB values ​​of a reference image before outputting the result.

[0217] <How to generate 3DLUT data> Next, the method for generating 3DLUT data in this embodiment will be explained using Figure 18.

[0218] In this embodiment, the number of 3DLUT data points is 33 × 33 × 33, each having 33 steps in the RGB values.

[0219] The grid points 1801 shown in Figure 18 represent the pixel values ​​of 3DLUT data placed on a 3D graph with the three primary colors, red (R), green (G), and blue (B), as axes.

[0220] Each RGB value in the 3DLUT data is generated using linear interpolation in the three-dimensional space shown in Figure 18, based on the RGB values ​​of 24 color patches acquired by the pixel value acquisition unit 307. A detailed explanation of linear interpolation is omitted as it is a general coordinate interpolation method.

[0221] In this embodiment, linear interpolation was used as the means for generating 3DLUT data, but the interpolation method is not limited to this, and other interpolation methods such as bilinear interpolation or bicubic interpolation may also be used.

[0222] Through the above process, the color matching application 301 can obtain the pixel values ​​of each color patch within the color chart 500, regardless of the tilt or rotation angle of the color chart 500 contained in the image.

[0223] [Second Embodiment] In the first embodiment, we described a method for calculating the position of each color patch by performing inference using a second learning model on an image extracted based on the inference results of a first learning model.

[0224] In this embodiment, a method for calculating the position of each color patch using only the inference results of the second learning model described in the first embodiment, with a simpler configuration, is described.

[0225] <Software Configuration> Figure 19 is a software configuration diagram of the communication device 200 in this embodiment.

[0226] When the reference selection image 804 in Figure 8(b) is tapped, the display control unit 302 requests the color patch position calculation unit 305 to acquire color patch position information, and obtains the position information of each color patch from the color patch position calculation unit.

[0227] The display control unit 302 generates an extracted image of the color chart 500 from the acquired position information of each color patch, and displays the extracted image of the color chart 500 in the selected thumbnail image display section 806. Other than the above, the description is the same as in the first embodiment and will be omitted.

[0228] <Processing to obtain position information of color patches> Figure 20 shows the processing flow for obtaining the cropped image and color patch position information 809 of the reference image 807 displayed in the selected thumbnail image display section 806 in this embodiment, and the cropped image and color patch position information 810 of the target image 808.

[0229] The processing flow shown in Figure 20 is a process that is executed in the color matching application 301 when the operation unit 206 detects a tap on the reference selection image 804 and the target selection image 805 while the thumbnail image display section 803 in Figure 8(b) is being displayed.

[0230] In S2001 of Figure 20, the color matching application 301 requests the color patch position calculation unit 305 to acquire color patch position information and inputs the paths of the reference selection image 804 and the target selection image 805.

[0231] In S2002 of Figure 20, the color matching application 301 acquires the color patch position information of the color chart 500 calculated by the color patch position calculation unit 305.

[0232] The method for calculating the positional information of the color patches is shown in Figure 15, so we will omit the explanation here.

[0233] In S2003 of Figure 20, the color matching application 301 generates a cropped image of the color chart 500 based on the color patch position information of the color chart 500 acquired in S2003.

[0234] In S2004 of Figure 20, the color matching application 301 displays the position information of the color patches acquired in S2002 and the cropped image of the color chart 500 generated in S2003 in the selected thumbnail image display section 806.

[0235] Through the above process, the color matching application 301 obtains the location information of the color patches.

[0236] <Processing by the detection position information acquisition unit 305> Next, using Figure 21, the processing flow when the detection position information acquisition unit 305 receives a request from the color patch position calculation unit 304 to acquire position information for multiple partial regions of the color chart 500 in this embodiment will be explained.

[0237] The processing flow shown in Figure 21 is a process that is executed in the color matching application 301 when the operation unit 206 detects a tap on the reference selection image 804 and the target selection image 805 while the thumbnail image display section 803 in Figure 8(b) is being displayed.

[0238] In S2101 of Figure 21, the detection position information acquisition unit 305 instructs the inference execution unit 312 to perform inference using a second learning model in order to acquire detection position information for multiple subregions of the color chart 500.

[0239] In step S2102 of Figure 21, the detection position information acquisition unit 305 acquires position information of multiple subregions within the color chart 500, which is the inference execution result of the second learning model, from the inference execution unit 312.

[0240] Through the above process, the detected position information acquisition unit 305 acquires position information for multiple sub-regions of the color chart 500 contained within the image.

[0241] [Other embodiments] Although the present invention has been described using the detection of color charts and the color patches contained therein as an example, it is not necessarily limited to this. For example, it can be used to detect parts containing a specific pattern in a factory, or to detect codes containing a specific pattern attached to packages in logistics.

[0242] Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or CPU or MPU, etc.) of that system or device read and execute the program.

Claims

1. An image processing device that acquires the pixel values ​​of multiple color patches included in a color chart within an image, An input means for inputting an image including a color chart on which the aforementioned color patches are arranged, A first identification means for identifying the locations of multiple subregions contained within a color chart in the input image, based on the results of inference by a learning model for detecting subregions containing the color patches in the input image, A second identification means for identifying the positions of the multiple color patches from the positional information of the multiple subregions, An acquisition means for acquiring the pixel values ​​of the plurality of color patches based on the position of the color patch, Equipped with, The image processing apparatus is characterized in that the learning model is one in which multiple subregions of the color chart are labeled.

2. The image processing apparatus according to claim 1, characterized in that the learning model detects a subregion including color patches located at least at the four corners of the color chart.

3. An image processing device that acquires the pixel values ​​of multiple color patches included in a color chart within an image, An input method for inputting an image including a color chart, A first identification means for identifying the position of the color chart within the input image based on the results of inference by a first learning model for detecting the entire area of ​​the color chart in the input image, A generation means that generates a cropped image by cutting out a region including the color chart from the input image based on the position of the color chart identified by the first identification means, A second identification means for identifying the positions of multiple subregions of the color chart in the input image, based on the results of inference by a second learning model for detecting subregions of the color chart in the cropped image, A third identification means for identifying the positions of the multiple color patches from the positional information of the multiple subregions, An acquisition means for acquiring the pixel values ​​of the plurality of color patches based on the position of the color patch, An image processing device having

4. The image processing apparatus according to claim 3, characterized in that the generation means generates a cropped image with a predetermined margin of size attached to the result of inference by the first learning model.

5. The image processing apparatus according to claim 3, characterized in that the second learning model labels a plurality of subregions for identifying the location of all color patches in the color chart.

6. The aforementioned color chart has the color patches arranged in an M x N pattern (where M and N are integers of 2 or more). The image processing apparatus according to claim 3, characterized in that the second learning model detects color patches located at least at the four corners of the color chart.

7. The third identification means obtains the respective central position of each color patch based on the inference results of the second learning model, The image processing apparatus according to claim 3, characterized in that the acquisition means acquires the average pixel value around the central position of the color patch.

8. The image processing apparatus according to claim 3, further comprising a display means for displaying the position of a color patch identified by the third identification means.

9. The input means inputs a first image including the color chart and a second image, The acquisition means acquires the pixel values ​​of the color patches of the first image and the pixel values ​​of the color patches of the second image. The image processing apparatus according to claim 3, further comprising LUT generation means for generating a LUT based on the pixel values ​​of the color patch of the first image and the pixel values ​​of the color patch of the second image.

10. The image processing apparatus according to claim 9, further comprising a display means for displaying an image obtained by applying the LUT generated by the LUT generation means to the second image.

11. A method for controlling an image processing device that acquires pixel values ​​of multiple color patches included in a color chart within an image, The input process involves entering an image that includes a color chart, A first identification step of identifying the position of the color chart within the input image based on the results of inference by a first learning model for detecting the entire area of ​​the color chart in the input image, A generation step that generates a cropped image by cutting out a region including the color chart from the input image based on the position of the color chart identified in the first specific step, A second identification step of identifying the locations of multiple subregions of the color chart in the input image based on the results of inference by a second learning model for detecting subregions of the color chart in the cropped image, A third identification step involves determining the positions of the multiple color patches from the positional information of the multiple subregions, An acquisition step of acquiring the pixel values ​​of the plurality of color patches based on the position of the color patch, A control method for an image processing apparatus having [a certain feature].

12. A computer-executable program that causes a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 10.

Images