Image analysis device, information processing method, and program

Abstract

This technology improves the accuracy of analysis of the imaged object. [Solution] An image analysis device according to one aspect of the present disclosure includes a control unit configured to acquire an image captured after being given instructions for the imaging composition, to provide the acquired image and instructions for the imaging composition to an analysis model, to have the analysis model perform image analysis, to acquire the image analysis results from the analysis model, and to output information regarding the analysis results of the acquired image.

Description

Technical Field

[0001] The present disclosure relates to an image analysis device, an information processing method, and a program.

Background Art

[0002] In Non-Patent Document 1, a system for understanding a document by a large language model has been proposed. In Non-Patent Document 1, visual information such as charts included in a document image is understood by a large language model together with text.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, the development of large-scale (visual) language models has advanced, and it has become easier to perform analysis on images. For example, according to a conventional method such as Non-Patent Document 1, an image having a predetermined configuration can be analyzed by giving the image to an analysis model. However, the inventor of the present invention has found that the conventional method has the following problems. That is, in the conventional method, the analysis model cannot obtain information that serves as a hint for analysis, such as the target to be focused on and how the target appears. If the identification of such information is incorrect, the accuracy of analysis by the analysis model may decrease.

[0005] In one respect, this disclosure has been made in consideration of these circumstances. One of the purposes of this disclosure is to provide a technology that improves the accuracy of the analysis of the imaged object. [Means for solving the problem]

[0006] This disclosure adopts the following configuration to solve the aforementioned problems. Note that the following configurations can be combined as appropriate.

[0007] An image analysis device relating to one aspect of this disclosure includes a control unit configured to acquire an image captured after being given instructions for the imaging composition, provide the acquired image and instructions for the imaging composition to an analysis model, have the analysis model perform image analysis, obtain the image analysis results from the analysis model, and output information regarding the analysis results of the acquired image. According to this configuration, in addition to the captured image, instructions for the imaging composition used when taking the image are also provided to the analysis model. This allows the analysis model to be given information that serves as a hint for the analysis. Therefore, according to this configuration, an improvement in the accuracy of the analysis of the image target can be expected.

[0008] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requiring the placement of a predetermined location of the object to be imaged according to a marker on the imaging screen. With this configuration, by clarifying the position of the object to be imaged, it is possible to expect an improvement in the accuracy of the analysis of the object to be imaged.

[0009] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that a physical marker be added to a predetermined location on the object to be imaged. According to this configuration, the imaging object By clearly defining the elephant's position, we can expect to improve the accuracy of the analysis of the imaged object.

[0010] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requiring that a predetermined portion of the object to be imaged be covered. With this configuration, by covering parts other than the object to be imaged, it is possible to expect an improvement in the accuracy of the analysis of the object to be imaged.

[0011] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction that requests that a predetermined part of the object to be imaged be placed at a predetermined position in the image. With this configuration, by clarifying the position of the object to be imaged, it is possible to expect an improvement in the accuracy of the analysis of the object to be imaged.

[0012] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that the object to be imaged be imaged in a predetermined orientation. With this configuration, by clarifying the orientation of the object to be imaged, it is possible to expect an improvement in the accuracy of the analysis of the object to be imaged.

[0013] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that the object to be imaged be imaged with a predetermined background color. With this configuration, by clearly distinguishing between the object to be imaged and parts that are not, it is possible to expect an improvement in the accuracy of the analysis of the object to be imaged.

[0014] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that the object to be imaged be imaged under predetermined lighting conditions. With this configuration, it is possible to improve the accuracy of the analysis of the object to be imaged by improving the visibility of the object to be imaged.

[0015] In an image analysis device relating to one aspect of this disclosure, the imaging composition instruction may be specified when the image is captured. This makes it possible to output different imaging composition instructions for each image. If the imaging composition instruction differs for each image, it becomes necessary to change the composition of the disguised object for each instruction, which is time-consuming. Therefore, this configuration can be expected to suppress the disguise of the object being imaged.

[0016] In an image analysis device relating to one aspect of this disclosure, specifying the imaging composition instruction when capturing an image may be configured by selecting one or more instruction candidates from a plurality of instruction candidates and specifying the selected one or more instruction candidates as the imaging composition instruction. This makes it possible to output a different imaging composition instruction for each image capture. Therefore, this configuration can be expected to suppress the falsification of the image target.

[0017] In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition includes one or more parameters, and specifying the instruction for the imaging composition when capturing an image may include specifying the value of each of the one or more parameters. This makes it possible to output a different instruction for the imaging composition for each image. With this configuration, it is expected that the falsification of the object to be imaged can be suppressed.

[0018] In an image analysis device relating to one aspect of this disclosure, the image contains a document as the object to be captured, and the image analysis may include identifying at least a part of the contents of the document. With this configuration, an improvement in the accuracy of the analysis of the object to be captured can be expected when analyzing a document.

[0019] In an image analysis device relating to one aspect of this disclosure, the image contains an article as the object to be captured, and the image analysis may include evaluating at least one of the condition and authenticity of the article. With this configuration, an improvement in the accuracy of the analysis of the object to be captured can be expected when evaluating the condition or authenticity of the article.

[0020] In an image analysis device relating to one aspect of this disclosure, the analysis of the image involves identifying the object being imaged. This configuration may include the following. With this configuration, an improvement in the accuracy of the analysis of the imaging target can be expected when identifying the imaging target.

[0021] Note that the embodiments of the present disclosure are not necessarily limited to the above image analysis apparatus. As another aspect of the image analysis apparatus according to each of the above aspects, one aspect of the present disclosure may be an image analysis method (control method) for realizing all or a part of each of the above configurations, or a program, or a machine-readable storage medium such as a computer that stores such a program. Here, the machine-readable storage medium may be a non-temporary medium that stores information such as a program by an electrical, magnetic, optical, mechanical, or chemical action. The non-temporary storage medium may include a storage medium (CD, DVD, semiconductor memory, etc.), an auxiliary storage device of a computer, an external storage device connected to the computer, and the like.

[0022] For example, the image analysis method according to one aspect of the present disclosure may be executed by a computer. The image analysis method may include obtaining an image captured upon receiving an instruction for an imaging composition, providing the obtained image and the instruction for the imaging composition to an analysis model, causing the analysis model to execute analysis of the image, obtaining an analysis result of the image from the analysis model, and outputting information regarding the analysis result of the obtained image.

[0023] For example, the program according to one aspect of the present disclosure may be a program for causing a computer to execute an image analysis method. The image analysis method may include obtaining an image captured upon receiving an instruction for an imaging composition, providing the obtained image and the instruction for the imaging composition to an analysis model, causing the analysis model to execute analysis of the image, obtaining an analysis result of the image from the analysis model, and outputting information regarding the analysis result of the obtained image.

Advantages of the Invention

[0024] According to one aspect of the present disclosure, the accuracy of analysis of an imaging object can be improved.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 is a diagram schematically showing an example of a scene to which the present disclosure is applied. [Figure 2] Figure 2 schematically shows an example of a marker on the imaging screen. [Figure 3] Figure 3 schematically shows an example of a marker on the imaging screen. [Figure 4] Figure 4 schematically shows an example of a marker on the imaging screen. [Figure 5] Figure 5 schematically illustrates an example of adding physical markers. [Figure 6] Figure 6 schematically shows an example of a scene in which a predetermined area of ​​the imaging target is covered. [Figure 7] Figure 7 schematically shows an example of a scenario in which a predetermined part of the image to be captured is positioned at a predetermined location in the image. [Figure 8] Figure 8 schematically shows an example of a scenario in which an object to be imaged is captured in a specified orientation. [Figure 9] Figure 9 schematically shows an example of a scenario in which the target object is imaged with a predetermined background color. [Figure 10] Figure 10 schematically shows an example of a scenario in which an object to be imaged is captured under predetermined illumination conditions. [Figure 11] Figure 11 schematically shows an example of a scenario in which instructions for the imaging composition are given. [Figure 12] Figure 12 schematically shows an example of a scene in which captured images are analyzed. [Figure 13] Figure 13 schematically shows an example of a scene in which captured images are analyzed. [Figure 14] Figure 14 schematically shows an example of a scene in which captured images are analyzed. [Figure 15] Figure 15 schematically shows an example of the hardware configuration of an image analysis device. [Figure 16] Figure 16 schematically shows an example of the software configuration of an image analysis device. [Figure 17] Figure 17 is a flowchart illustrating an example of the processing procedure of this disclosure. [Figure 18] Figure 18 shows the prompts entered in the comparative example. [Figure 19] Figure 19 shows the prompts entered in the example. [Figure 20]Figure 20 shows an example of a sample image input in the example. [Figure 21] Figure 21 shows the extraction results for the comparative example and the example. [Modes for carrying out the invention]

[0026] Hereinafter, embodiments relating to one aspect of this disclosure will be described with reference to the drawings. However, the embodiments described below are merely illustrative in all respects of this disclosure. Various improvements or modifications may be made without departing from the scope of this disclosure. In implementing this disclosure, specific configurations may be adopted as appropriate depending on the embodiment. In this embodiment, the data appearing is described in natural language, but more specifically, it is specified in pseudo-language, commands, parameters, machine code, electrical signals, etc., that can be recognized by machines such as computers.

[0027] §1 Examples of Application Figure 1 is a schematic diagram illustrating an example of a scenario to which this disclosure applies. The image analysis device 1 according to this embodiment is one or more computers configured to output information 60 regarding the analysis result 50 of an image (captured image 30) captured in response to an instruction 20 for the imaging composition. In one example, the image analysis device 1 may be a server device. In another example, the image analysis device 1 may be a user terminal.

[0028] In this embodiment, the image analysis device 1 provides user U with an instruction 20 for the imaging composition. User U obtains an image 30 by imaging the target IT with the imaging device C according to the instruction 20 for the imaging composition. The image analysis device 1 acquires the image 30 from user U. The image analysis device 1 provides the instruction 20 for the imaging composition and the image 30 to the analysis model 40, causing the analysis model 40 to perform analysis of the image 30. The image analysis device 1 acquires the analysis result 50 of the image 30. The image analysis device 1 then outputs information 60 related to the acquired analysis result 50 of the image 30.

[0029] According to this embodiment, the image analysis device 1 provides the analysis model 40 with not only the captured image 30 but also instructions 20 regarding the imaging composition. This allows the analysis model 40 to be provided with information that serves as a hint for the analysis. As a result, an improvement in the accuracy of the analysis of the imaged object can be expected.

[0030] [Target of imaging] The imaged IT can be any object that can be analyzed by the analysis model 40, such as a document, a person, or an object.

[0031] [Instructions for the composition of the image] The imaging composition indicates how the target IT will be displayed in the image. Therefore, the imaging composition instruction 20 may be an instruction given to the user U in the scene of imaging the target IT, specifying how the target IT will be displayed in the image. The imaging composition instruction 20 is not particularly limited and may be set as appropriate, as long as the user U can image the target IT according to the instruction. For example, the imaging composition instruction 20 may be configured to include information that can serve as a hint for the analysis of the target IT by the analysis model 40. The imaging composition instruction 20 may also be configured in a way that is understandable to the user U in natural language.

[0032] The imaging composition instruction 20 may include an instruction to limit the range to be analyzed by the analysis model 40. This improves the accuracy of the analysis of the captured image 30 by the analysis model 40.

[0033] In one example, the imaging composition instruction 20 may be an instruction requesting that a predetermined location of the imaging target IT be positioned according to a marker on the imaging screen. The marker on the imaging screen may be configured to be displayed superimposed on the imaging target IT. The shape, size, etc. of the marker displayed on the imaging screen are not particularly limited and may be appropriately defined depending on the embodiment. Furthermore, the marker is used in imaging It may be fixed and displayed at any position on the screen. The designated location of the imaging target IT may be set as appropriate depending on the type of imaging target IT and the purpose of the analysis.

[0034] The designated area may be the entirety of the IT being imaged, or a part of it. For example, if the designated area is a part of the IT being imaged, it may be an area that is analyzed intensively in the analysis model 40. For example, if the IT being imaged is a document, the designated area may be a specified description within the document. Note that the designated area of ​​a document may also include descriptions on the back. If the IT being imaged is an item, the designated area may be an area that is particularly important in the evaluation of the item. For example, an important area may include an area that is prone to damage to the item (such as the corner of a card). If the IT being imaged is a person, the designated area may be an area that is effective for person identification. Typically, the designated area for a person may be the face.

[0035] As a result of imaging, the markers on the imaging screen may or may not be displayed in the captured image 30. If the markers are not displayed, the captured image 30 may be configured to retain information indicating the position of the markers. Accordingly, the analysis model 40 may be provided with information indicating the position of the markers in addition to the imaging composition instruction 20.

[0036] Figures 2 to 4 schematically show examples of markers (SM1, SM2, SM3) on the imaging screen IS. A receipt is selected as the imaging target IT1, and the scenario assumes that an image is being captured to analyze the contents of the receipt. In this example, the section labeled "TOTAL" is set as the designated area PL. In one example, the marker may be a marker SM1 that points to a single point on the imaging screen IS, as shown in Figure 2. In this case, for example, the user U may be presented with a statement such as "Please align the marker on the screen with 'TOTAL' on the receipt and take the picture" as the imaging composition instruction 20. In another example, the marker may be a straight line SM2 placed at both the left and right ends of the imaging screen IS, as shown in Figure 3. In this case, for example, the user U may be presented with a statement such as "Please align the height of the line on the screen with the position of 'TOTAL' and take the picture" as the imaging composition instruction 20. In yet another example, the marker may be a straight line SM3 connecting both the left and right ends of the imaging screen IS, as shown in Figure 4. In this case, for example, the user U may be presented with a sentence such as "Please take the picture so that 'TOTAL' is positioned below the line on the screen" as instruction 20 for the imaging composition.

[0037] For example, the imaging composition instruction 20 may be an instruction requesting that a physical marker be added to a predetermined location on the imaging target IT. For example, adding a physical marker may be the placement of any object. The imaging composition instruction 20 may include an instruction to specify a particular object as the physical marker. In this case, the object serving as the physical marker may be arbitrarily selected to the extent that it does not affect the analysis of the imaging target IT (e.g., a pen, fingertip). Alternatively, the imaging composition instruction 20 may not include an instruction to specify a particular object as the physical marker. In this case, the image analysis device 1 (analysis model 40) may identify the physical marker by extracting the object arbitrarily selected and placed by the user U from the captured image 30. The predetermined location on the imaging target IT may be set appropriately according to the type of imaging target IT and the purpose of the analysis, similar to the case where a marker is displayed on the imaging screen.

[0038] Figure 5 schematically shows an example of a scenario in which a physical marker is added. A receipt is selected as the image target IT1, and the scenario assumes the placement of a marker when capturing an image to analyze the contents of the receipt. In this example, the section labeled "TOTAL" is set as the designated location PL. In one example, the marker may be a pen PM, as shown in Figure 5. In this case, for example, the user U may be presented with a sentence such as "Align the pen tip with the section corresponding to 'TOTAL' and take the picture" as the imaging composition instruction 20. The above example shows a case where a physical marker is specified. Otherwise, for example... For example, as instruction 20 for the imaging composition, a sentence such as "Place a marker in the area corresponding to 'TOTAL' and take the picture" may be presented to user U.

[0039] Furthermore, adding physical markers may involve applying a predetermined process to the imaging target IT, to the extent that it does not affect the analysis. For example, adding physical markers may include forming markers on the imaging target IT by writing.

[0040] For example, the imaging composition instruction 20 may be an instruction requesting that a predetermined area of ​​the imaging target IT be covered. For example, covering may consist of obscuring at least a part of the predetermined area by placing an object. The imaging composition instruction 20 may include an instruction specifying a particular object as the covering object. In this case, any object such as a hand or a piece of paper may be selected as the covering object. Alternatively, the imaging composition instruction 20 may not include an instruction specifying a particular object as the covering object. In this case, the image analysis device 1 (analysis model 40) may identify the covering object by extracting the object arbitrarily selected and placed by the user U from the captured image 30. The predetermined area to be covered may be set appropriately depending on the type of imaging target IT and the purpose of the analysis. For example, the predetermined area to be covered may be any part that is not the target of analysis.

[0041] Furthermore, covering may include covering unwanted parts by applying a predetermined process to the captured image 30. For example, the predetermined process may include filling in, mosaic processing, blurring, etc., for unwanted parts of the captured image 30.

[0042] Figure 6 schematically shows an example of a scenario in which a predetermined area UP of the image target IT is covered. A receipt is selected as the image target IT1, and the scenario assumes that unnecessary parts are covered when capturing an image to analyze the contents written on the receipt. In this example, the predetermined area UP may be any part that is not the target of analysis. For example, when analyzing the part that says "TOTAL", the user U may be presented with a statement such as "Please capture the image while hiding everything except the part that corresponds to 'TOTAL'" as the image composition instruction 20. Alternatively, the user U may be presented with a statement such as "Please capture the image while hiding any irrelevant or unnecessary information" as the image composition instruction 20 without explicitly specifying the target of analysis.

[0043] For example, the imaging composition instruction 20 may be an instruction requesting that a predetermined location of the target IT be placed at a predetermined position on the image (imaging image 30). The predetermined location may be arbitrarily set to a position such as the top / bottom of the image, the center, or the top / bottom edge. The predetermined location may also be the entire image. For example, if the imaging image 30 is captured as a magnified image of a predetermined location of the target IT, and as a result the predetermined location of the target IT is captured in the entire image, the predetermined location may be the entire image. The predetermined location of the target IT may be set appropriately according to the type of target IT and the purpose of the analysis, similar to when displaying a marker on the imaging screen.

[0044] Figure 7 schematically shows an example of a scenario in which a predetermined location of the image target IT is positioned at a predetermined location in the image. A receipt is selected as the image target IT1, and the scenario assumes that an image is to be captured for analysis of the contents written on the receipt. In this example, the top edge of the receipt is set as the predetermined location. The predetermined location is also set as the top edge on the image (image capture screen IS). In this case, for example, a statement such as "Please align the top edge of the receipt with the top edge of the image and take the picture." may be presented to the user U as an instruction 20 for the image composition.

[0045] Furthermore, the imaging composition instruction 20 may include instructions to improve the identifiability of the object to be analyzed. This can improve the accuracy of the analysis of the captured image 30 by the analysis model 40.

[0046] In one example, the imaging composition instruction 20 requests that the imaging target IT be imaged in a predetermined orientation. The instruction may be a general one. The predetermined orientation may be specified as appropriate depending on the imaging target IT. For example, if the imaging target IT is a document, the predetermined orientation may be specified so that the target items are in the correct orientation. Also, if the imaging target IT is an object, the orientation may be specified according to the object, such as the front, side, or top.

[0047] Figure 8 schematically shows an example of a scenario in which an image target IT is captured in a specified orientation. The image target IT2 is a receipt containing multiple orientations of text, and the scenario assumes the capture of an image for analyzing the contents of the receipt. In this example, the predetermined orientation is set such that the part containing "Receipt" is upright (correctly oriented). In this case, for example, the user U may be presented with an instruction 20 for the imaging composition such as, "Please take the image so that the part corresponding to 'Receipt' is in the correct orientation."

[0048] In one example, the imaging composition instruction 20 may be an instruction requesting that the imaging target IT be imaged with a predetermined background color. The predetermined background color may be any background color that can distinguish it from the imaging target IT. For example, the predetermined background color may be a single color (or a color close to a single color).

[0049] Figure 9 schematically shows an example of a scenario in which an object to be imaged (IT) is imaged with a predetermined background color (BG). A trading card is selected as the object to be imaged (IT3), and the scenario assumes that images are being taken to analyze the trading card. In this example, black is specified as the predetermined background color (BG). In this case, for example, a statement such as "Please take the image with a black background" may be presented to the user U as an instruction 20 for the imaging composition.

[0050] For example, the imaging composition instruction 20 may be an instruction requesting that the imaging target IT be imaged under predetermined lighting conditions. The predetermined lighting conditions may include requirements related to the appearance of the image, such as illumination from a specified angle, suppression of overexposure, and the addition of shadows.

[0051] Figure 10 schematically shows an example of a scenario in which an image target IT is captured under predetermined lighting conditions. A trading card is selected as the image target IT3, and the scenario assumes the capture of an image for analysis of the trading card. In this example, the predetermined lighting condition is illumination from the right. In this case, for example, a statement such as "Please illuminate from the right and take the picture" may be presented to the user U as instruction 20 for the imaging composition.

[0052] (How to specify instructions) The imaging composition instruction 20 may be specified at any time. In one example, the imaging composition instruction 20 may be a fixed instruction specified in advance. In this case, the specified imaging composition instruction 20 may be stored in the memory resources of the image analysis device 1. In another example, the imaging composition instruction 20 may be specified when the image 30 is captured. That is, the imaging composition instruction 20 may be specified when user U starts the imaging device C.

[0053] Figure 11 schematically shows an example of a scenario in which an imaging composition instruction 20 is specified. In one example, specifying an imaging composition instruction 20 when capturing an image 30 may be done by selecting one or more instruction candidates from a plurality of instruction candidates 200 and specifying the selected instruction candidate as the imaging composition instruction 20. The instruction candidates 200 may be predetermined instructions according to the analysis content and may be stored in the memory resources of the image analysis device 1. The method of selecting the instruction to be the imaging composition instruction 20 from the instruction candidates 200 is not particularly limited and may be defined as appropriate. For example, the selection method may be a method of random selection from the instruction candidates 200, a method of sequential selection, etc.

[0054] The imaging composition instruction 20 may be configured to include one or more parameters 250. Accordingly, specifying the imaging composition instruction 20 when capturing the captured image 30 may be done by specifying the value of one or more parameters 250. The parameters 250 may be predetermined according to the content of the imaging composition instruction 20. For example, the parameters 250 may include the position of the marker on the imaging screen IS, the position of the imaging target IT, the background color BG, etc. The method for specifying the parameters 250 may be set as appropriate, such as a method of probabilistically determining the parameter values ​​or a method of selecting from multiple parameter candidates.

[0055] Furthermore, the method for specifying the imaging composition instruction 20 may be a combination of the above-described methods. That is, the instruction candidates 200 may include instruction candidates that include the parameter 250. Accordingly, specifying the imaging composition instruction 20 may consist of selecting one or more instruction candidates from among the multiple instruction candidates 200, and specifying the value of the parameter 250 included in the selected instruction candidate.

[0056] (How to give instructions to the user regarding the image composition) The imaging composition instruction 20 may be given to user U from the image analysis device 1, or it may be given to user U via another device. If it is given via another device, the image analysis device 1 may obtain the imaging composition instruction 20 from that other device. For example, the image analysis device 1 may be a server device and the other device may be a user terminal, in which case the image analysis device 1 may obtain the imaging composition instruction 20 from the user terminal.

[0057] The imaging composition instruction 20 may be presented to the user U at any time before imaging. In one example, the imaging composition instruction 20 may be presented before the imaging device C is started up. In another example, the imaging composition instruction 20 may be presented simultaneously with the start up of the imaging device C.

[0058] [Analysis Model] The configuration of the analysis model 40 is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it has the ability to perform analysis based on the given imaging composition instruction 20 and the captured image 30. In one example, the analysis model 40 may be composed of a trained machine learning model. The configuration (type, structure, etc.) of the machine learning model may be determined arbitrarily. The trained machine learning model may include a large-scale generative model such as a large-scale visual language model (VLM). The large-scale generative model used in the analysis model 40 may be a multimodal model.

[0059] The analysis model 40 may be stored in the memory resources of the image analysis device 1, or it may be stored in an external computer. If the analysis model 40 is stored in an external computer, the image analysis device 1 may transmit the imaging composition instruction 20 and the captured image 30 to the external computer via a network and obtain the analysis result 50.

[0060] Figures 12 to 14 schematically show an example of analysis performed by the analysis model 40. The content of the analysis performed by the analysis model 40 is not particularly limited as long as it obtains any information related to the imaging target IT, and may be appropriately selected depending on the embodiment. In this case, the analysis model 40 may be configured as a dedicated model that has been trained to perform a specific analysis.

[0061] As shown in Figure 12, the imaged IT may be a document. That is, the captured image 30 may contain a document. Accordingly, the analysis of the captured image 30 by the analysis model 40 may include identifying at least a portion of the document's content. In this case, the analysis model 40 may be trained to identify the document's content. The document may be any document containing text. For example, identifying the document's content may include content identification such as receipt analysis or ingredient label analysis.

[0062] As shown in Figure 13, the object to be imaged IT may be an object. That is, the imaged image 30 The image may contain an object. Accordingly, the analysis of the captured image 30 by the analysis model 40 may include evaluating at least one of the condition and authenticity of the object. In this case, the analysis model 40 may be trained to evaluate the condition / authenticity of the object. The object may be a tangible object that can be evaluated. For example, the object may be trading cards, branded goods, clothing, etc., which can be evaluated for condition / authenticity at the time of transaction. The object may also be infrastructure equipment, defective goods, etc., which can be evaluated for condition. Condition evaluation may consist of estimating the condition of the object. Specifically, condition evaluation may consist of estimating the degree of damage, soiling, breakage, defects, deformation, wear, etc. of the object. Authenticity evaluation may consist of determining whether the object is genuine or not.

[0063] As shown in Figure 14, the imaging target IT may be any object, including people and objects. Accordingly, the analysis of the imaging target IT by the analysis model 40 may include identifying the imaging target IT. In this case, the analysis model 40 may be pre-trained to identify the imaging target IT. For example, identification may include identity verification, person identification, category determination, attribute determination, etc.

[0064] (How to provide instructions for the imaging composition to the analysis model) The imaging composition instruction 20 given to the analysis model 40 may or may not be the same as the one given to user U. That is, the imaging composition instruction 20 may be appropriately converted to indicate its meaning before being given to the analysis model 40. In one example, an informational statement indicating the content of the imaging composition instruction 20 may be generated by manual input by an operator of the image analysis device 1. In another example, an informational statement indicating the content of the imaging composition instruction 20 may be generated according to a pre-prepared template. The image analysis device 1 may provide the analysis model 40 with the said informational statement. The format of the above informational statement is not particularly limited and may be defined as appropriate.

[0065] After acquiring the captured image 30, the image analysis device 1 may determine whether the captured image 30 was captured according to the imaging composition instruction 20 before providing the captured image 30 to the analysis model 40. The image analysis device 1 may perform this determination using an identification model separate from the analysis model 40. The identification model may be stored in the memory resources of the image analysis device 1 or on an external computer. If the identification model is stored on an external computer, the image analysis device 1 may transmit the imaging composition instruction 20 and the captured image 30 to the external computer via a network and have the determination performed. The image analysis device 1 may obtain the determination result from the external computer. If it is determined that the image was not captured according to the imaging composition instruction 20, the image analysis device 1 may give the imaging composition instruction 20 to the user U again and obtain the captured image 30. The image analysis device 1 may give the imaging composition instruction 20 to the user U and instruct them to take images until it is determined that the image was captured according to the imaging composition instruction 20.

[0066] The image analysis device 1 obtains an analysis result 50 by providing the imaging composition instruction 20 and the captured image 30 to the analysis model 40. The information provided to the analysis model 40 may include information other than the imaging composition instruction 20 and the captured image 30. For example, the image analysis device 1 may provide the analysis model 40 with instructions regarding the output format of the analysis result 50. In one example, the image analysis device 1 may provide instructions to output the analysis result 50 in natural language. In another example, the image analysis device 1 may provide instructions to output the analysis result 50 in a predetermined file format (such as JSON).

[0067] [Outputting information about the analysis results] In a typical example, outputting information 60 related to the analysis result 50 may be done by outputting the analysis result 50 as is. However, the content of information 60 is not limited to this example, as long as it relates to the analysis result 50, and may be determined appropriately depending on the embodiment. In another example, the image analysis device 1 may perform arbitrary information processing on the analysis result 50. Outputting information 60 related to the analysis result 50 may also be done together with the analysis result 50. This may be configured by outputting the result of this information processing instead of the analysis result 50.

[0068] §2 Example Configuration [Hardware configuration] Figure 15 schematically shows an example of the hardware configuration of the image analysis device 1 according to this embodiment. In one example, the image analysis device 1 may be configured as a computer in which a control unit 11, a storage unit 12, an external interface 13, a communication interface 14, an input device 15, and an output device 16 are electrically connected.

[0069] The control unit 11 is configured to perform information processing based on the program and various data. For example, the control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). Good. The control unit 11 (CPU) is an example of a processor resource.

[0070] The storage unit 12 is configured to hold arbitrary data. For example, the storage unit 12 may include a hard disk drive, a solid-state drive, a semiconductor memory, etc. The storage unit, RAM, and ROM are examples of memory resources. In one example, the storage unit 12 may store a program 81. The program 81 is a program that causes the image analysis device 1 to execute the information processing (Figure 17, described later) related to this disclosure. The program 81 includes a series of instructions for the information processing.

[0071] In one example, program 81 may be stored in storage medium 91 instead of or together with storage unit 12. Storage medium 91 is configured to store various types of information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. Storage unit 12 and storage medium 91 are examples of non-temporary storage media. Image analysis device 1 may retrieve program 81 from storage medium 91. Storage medium 91 may be a disk-type storage medium (CD, DVD, etc.) or a non-disk-type storage medium such as semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in storage medium 91. The type of drive device may be selected according to the storage medium 91. The drive device may be connected to image analysis device 1 by any method. Storage medium 91 may include an external storage device that can be connected to image analysis device 1.

[0072] The external interface 13 may be configured as appropriate to connect to an external device via wired or wireless connection, for example, by a USB (Universal Serial Bus) port, a dedicated port, etc. The communication interface 14 is configured to perform wired or wireless communication over a network. The communication interface 14 may be configured as, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc. The network standard is not particularly limited and may be selected as appropriate depending on the embodiment. For example, the type of network may be selected as appropriate from the Internet, wireless communication network, mobile communication network, telephone network, dedicated network, etc. The image analysis device 1 may perform data communication with other computers via the communication interface 14.

[0073] The input device 15 is configured to accept information input. The input device 15 may consist of, for example, a camera, microphone, mouse, keyboard, touch panel, or operator. The output device 16 is configured to output information. The output device 16 may consist of, for example, a display or speaker. The image analysis device 1 may be operated using the input device 15 and the output device 16. The input device 15 and the output device 16 may be directly connected to the image analysis device 1, or they may be indirectly connected via at least one of the external interface 13 and the communication interface 14. The output device 16 may be integrated in at least part with a touch panel display or the like.

[0074] Regarding the specific hardware configuration of the image analysis device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (Field-Programmable Gate Arrays), DSPs (Digital Signal Processors), and ECUs. It may consist of an Electronic Control Unit (ECC), a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), etc. Program 81 may be stored in an external storage device such as a Network Attached Storage (NAS). An external storage device is also an example of a non-temporary storage medium.

[0075] [Software Configuration] Figure 16 schematically shows an example of the software configuration of the image analysis device 1 according to this embodiment. The control unit 11 of the image analysis device 1 executes instructions contained in the program 81 stored in the storage unit 12 using the CPU. As a result, the image analysis device 1 operates as a computer equipped with an instruction provision unit 111, an image acquisition unit 112, an analysis processing unit 113, and an output processing unit 114 as software modules. In other words, in one example, each software module of the image analysis device 1 may be implemented by the control unit 11 (CPU).

[0076] The instruction provision unit 111 is configured to generate an instruction 20 for the imaging composition and provide it to user U. The image acquisition unit 112 is configured to acquire the image 30 captured by user U. The analysis processing unit 113 is configured to provide the instruction 20 for the imaging composition and the image 30 to the analysis model 40 and acquire the analysis result 50. The output processing unit 114 is configured to output information 60 related to the analysis result 50.

[0077] In this example of the embodiment, each software module of the image analysis device 1 is implemented by a general-purpose CPU. However, the method of implementing each of the above modules is not limited to this example and may be modified as appropriate depending on the embodiment. Some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. Regarding the software configuration of the image analysis device 1, modules may be omitted, replaced, and added as appropriate depending on the embodiment.

[0078] §3 Example of Operation Figure 17 is a flowchart showing an example of the processing procedure by the image analysis device 1 according to this embodiment. The control unit 11 of the image analysis device 1 executes instructions contained in the program 81 held in the storage unit 12. As a result, the image analysis device 1 operates as a computer capable of performing the following information processing. The control unit 11 is configured to perform the following information processing. The following processing procedure is an example of an information processing method performed by a computer. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0079] (Step S101) In step S101, the control unit 11 acts as an instruction provider 111 and gives the user U an instruction 20 for the imaging composition. In one example of analysis, the instruction 20 for the imaging composition may be a fixed instruction specified in advance. In another example, the instruction 20 for the imaging composition may be specified when the user U takes the image 30. Once the instruction 20 for the imaging composition is given to the user U, the control unit 11 proceeds to the next step 102.

[0080] (Step S102) In step S102, the control unit 11 operates as an image acquisition unit 112 and acquires the captured image 30 taken by user U. Once the captured image 30 is acquired, the control unit 11 proceeds to the next step S103.

[0081] After acquiring the captured image 30, the control unit 11 may determine whether the captured image 30 was captured according to the imaging composition instruction 20. This determination may be performed using an arbitrary identification model. The identification model may be stored in the storage unit 12 or in an external computer. If it is determined that the image was not captured according to the imaging composition instruction 20, the control unit 11 may return to step S101, give the user U the imaging composition instruction 20 again, and instruct them to capture the image. The control unit 11 may repeatedly execute steps S101 and S102 until it is determined that the captured image 30 was captured according to the imaging composition instruction 20.

[0082] (Step S103) In step S103, the control unit 11 operates as an analysis processing unit 113, providing the imaging composition instruction 20 and the captured image 30 to the analysis model 40, and obtaining the analysis result 50. The analysis model 40 may be composed of a trained machine learning model. The analysis model 40 may be stored in the storage unit 12 or on an external computer. If the analysis model 40 is stored on an external computer, the control unit 11 may transmit the imaging composition instruction 20 and the captured image 30 to the external computer via the network and obtain the analysis result 50. The imaging composition instruction 20 given to the analysis model 40 may or may not be the same as the one given to user U. Once the analysis result 50 is obtained, the control unit 11 proceeds to the next step S104.

[0083] (Step S104) In step S104, the control unit 11 operates as an output processing unit 114 and outputs information 60 related to the analysis result 50. In one example, the control unit 11 may output the analysis result 50 as is. In another example, the control unit 11 may perform arbitrary information processing on the analysis result 50. The control unit 11 may output the result of this information processing together with the analysis result 50 or in place of the analysis result 50.

[0084] The destination of the output information 60 is not particularly limited and may be appropriately selected depending on the embodiment. The output destination may be, for example, RAM, storage unit 12, output device 16, storage medium 91, external computer, external storage device, etc. When the output of information 60 is complete, the control unit 11 terminates the processing procedure related to this example of operation.

[0085] §4 Variant While embodiments of this disclosure have been described in detail above, the above description is merely illustrative in all respects of this disclosure. Needless to say, various improvements or modifications can be made without departing from the scope of this disclosure. The processes and means described in this disclosure can be freely combined and implemented, as long as no technical inconsistencies arise.

[0086] §5 Experimental Examples To verify that the accuracy of the analysis of the image target (analysis target) can be improved by providing instructions for the imaging composition to the analysis model, the following experiment was conducted. However, this disclosure is not limited to the following experimental example.

[0087] First, the analysis will focus on 973 SROIE (Scanned Receipts OCR and Information) cases. A dataset (Extraction) was prepared. The receipt images in the above dataset contain the following information. The task of extracting company names was performed on the analysis model using both the comparative example (standard input method) and the example (input method based on this disclosure) input methods. The results of extracting company names using the two methods described above were obtained. The analysis model used was GPT-4o (OpenAI).

[0088] Figure 18 shows the prompts entered in the comparative example. In the comparative example, the input was an instruction to extract a receipt image and the company name. Figure 19 shows the prompts entered in the embodiment. In the embodiment, in addition to the instruction to extract a receipt image and the company name, contextual information of the instruction (information about the imaging composition) was provided as input. Figure 20 shows an example of a sample image entered in the embodiment. As shown in Figure 20, a cross marker was placed near the location where the company name is written in the receipt image. The contextual information for the embodiment stated that a visual marker was placed near the company name.

[0089] Figure 21 shows the extraction results for the comparative example and the example. The extraction results were classified into exact match, partial match, and no match. An exact match was defined as the extracted string exactly matching the correct company name. A partial match was defined as the extracted string matching the correct company name to a predetermined degree of similarity or higher. No match was defined as not meeting the criteria for an exact match or partial match. As shown in Figure 21, the exact match rate for the comparative example was 80.27%. In contrast, the exact match rate for the example was 85.20%. Furthermore, the partial match rate for the comparative example was 15.82%. In contrast, the partial match rate for the example was 13.98%. In addition, the no match rate for the comparative example was 3.91%. In contrast, the no match rate for the example was 0.82%. In other words, by using an input like that of the example, the rates of partial matches and no matches decreased, and the rate of exact matches increased by the same amount. These results show that providing the analysis model with information about the imaging composition can improve the accuracy of the analysis of the imaged object. [Explanation of symbols]

[0090] 1. Image analysis device, 11. Control unit, 12. Storage unit, 13. External interface, 14. Communication interface, 15. Input device, 16. Output device, 81...Programs, 91...Storage media, 20. Instructions for imaging composition, 30. Captured image, 40...Analysis model, 50...Analysis results, 60...Information, 200... Candidate instructions, 250... Parameters, U...User, C...Imaging device, IT...Imaging target

Claims

1. The image is captured after being given instructions for the imaging composition. The acquired image and the instructions for the imaging composition are given to the analysis model, and the analysis model is made to perform the analysis of the image, thereby obtaining the analysis results of the image from the analysis model. Outputs information regarding the analysis results of the acquired image. It includes a control unit configured as follows: Image analysis device.

2. The aforementioned instruction for the imaging composition is an instruction requesting that the predetermined location of the object to be imaged be positioned according to the markers on the imaging screen. The image analysis apparatus according to claim 1.

3. The aforementioned instruction for the imaging composition is an instruction requesting that a physical marker be added to a predetermined location on the object to be imaged. The image analysis apparatus according to claim 1.

4. The instruction for the imaging composition is an instruction to cover a predetermined area of ​​the object to be imaged. The image analysis apparatus according to claim 1.

5. The instruction for the imaging composition is an instruction requesting that a predetermined part of the object to be imaged be placed at a predetermined position in the image. The image analysis apparatus according to claim 1.

6. The image analysis apparatus according to claim 1, wherein the instruction for the imaging composition is an instruction to require imaging the object to be imaged in a predetermined orientation.

7. The aforementioned instruction for the imaging composition is an instruction requesting that the object to be imaged be imaged with a predetermined background color. The image analysis apparatus according to claim 1.

8. The instruction for the aforementioned imaging composition is an instruction to image the target object under predetermined lighting conditions. The image analysis apparatus according to claim 1.

9. The instruction for the imaging composition is specified when the image is captured. The image analysis apparatus according to claim 1.

10. Specifying the imaging composition instructions when capturing the aforementioned image is performed by selecting one or more instruction candidates from a plurality of instruction candidates and specifying the selected one or more instruction candidates as the imaging composition instructions. The image analysis apparatus according to claim 9.

11. The instruction for the imaging composition includes one or more parameters, Specifying the imaging composition when capturing the aforementioned image includes specifying the value of each of the one or more parameters. The image analysis apparatus according to claim 9.

12. The aforementioned image shows a document as the subject of the image capture. The image analysis apparatus according to claim 1, wherein the analysis of the image includes identifying at least a portion of the contents of the document.

13. The aforementioned image shows an object as the subject of the image, The analysis of the aforementioned image includes evaluating at least one of the condition and authenticity of the article. The image analysis apparatus according to claim 1.

14. The analysis of the aforementioned image includes identifying the object being imaged. The image analysis apparatus according to claim 1.

15. The analysis model is composed of a large-scale generative model, The image analysis apparatus according to claim 1.

16. A computer-based image analysis method, The image is captured after being given instructions for the imaging composition. The acquired image and the instructions for the imaging composition are given to the analysis model, and the analysis model is made to perform the analysis of the image, thereby obtaining the analysis results of the image from the analysis model. Outputs information regarding the analysis results of the acquired image. Including, Image analysis methods.

17. A program that causes a computer to perform an image analysis method, The aforementioned image analysis method is The image is captured after being given instructions for the imaging composition. The acquired image and the instructions for the imaging composition are given to the analysis model, and the analysis model is made to perform the analysis of the image, thereby obtaining the analysis results of the image from the analysis model. Outputs information regarding the analysis results of the acquired image. Including, program.

Images