Information processing apparatus, information processing method, and program

Abstract

To provide an information processor, an information processing method and a program for accurately specifying more character strings related to an object from an object image.SOLUTION: In an information processing device, a division unit divides a target image including a target object and a plurality of character strings into a plurality of small regions. The recognition unit performs a character recognition process using the target image, recognizes a plurality of character strings, and recognizes positions of the plurality of character strings in the target image. The index assignment unit assigns, to each of the plurality of small regions, an index associated with a relative positional relationship between the plurality of small regions in the target image. The input data generation unit generates input data including an input feature amount in which a position feature amount obtained by encoding an index of a small region corresponding to a position of a character string is added to a word feature amount extracted from each of a plurality of character strings. The output data acquisition unit acquires output data obtained by inputting the input data to the language model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] For example, Patent Document 1 describes an image processing device that extracts information from images such as flyer images. This image processing device includes a small region extraction unit, a character / object region extraction unit, a character recognition unit, and an object recognition unit.

[0003] The small region extraction unit described in Patent Document 1 extracts small region images from the entire flyer image. In Patent Document 1, the small region image is an area within the image surrounded by a boundary line, and is an image in which an image of an object as an object (hereinafter referred to as an object image) and an image of character strings are respectively drawn. The character / object region extraction unit described in Patent Document 1 extracts an image in which character strings are drawn from the image of the small region extracted by the small region extraction unit. Furthermore, the character / object region extraction unit extracts the object image from the small region extracted by the small region extraction unit.

[0004] The character recognition unit described in Patent Document 1 recognizes characters from each image based on the images of the characters that make up a character string included in the region of the character string image extracted by the character / object region extraction unit. The object recognition unit described in Patent Document 1 recognizes an object based on an object image included in the image of a small region extracted by the small region extraction unit. In Patent Document 1, "recognizing an object" means determining the name of the object included in the object image.

[0005] According to the description in Patent Document 1, it is possible to match the character information contained in an image with the name of the object, and by obtaining the product name recognized from the image of a product featured in a flyer image, it is possible to verify the information extracted by character recognition, thereby enabling more accurate information extraction. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-49943 Summary of the Invention [Problem to be solved by the invention]

[0007] According to the technology described in Patent Document 1, even if it is possible to associate a character string included in an image of the same small region with an object, it is difficult to associate a character string included in an image of a different small region with an object. Therefore, with the technology described in Patent Document 1, it is difficult to accurately identify many character strings related to an object from an image of the object that includes the object and multiple character strings.

[0008] One of the objectives of the present disclosure is to accurately identify as many character strings as possible that are related to an object from a target image. [Means for solving the problem]

[0009] According to one aspect of the present disclosure, there is provided an information processing device, A dividing means for dividing a target image including an object and a plurality of character strings into a plurality of small regions; a recognition unit that performs character recognition processing using the target image to recognize the plurality of character strings and recognizes positions of the plurality of character strings in the target image; an index assigning means for assigning an index to each of the plurality of small regions, the index being associated with a relative positional relationship of the plurality of small regions in the target image; an input data generation means for generating input data including input features obtained by adding position features obtained by encoding the indexes of the small regions corresponding to positions of the character strings to word features extracted from each of the plurality of character strings; and an output data acquisition means for acquiring output data obtained by inputting the input data into a language model.

[0010] An information processing method according to one aspect of the present disclosure includes: One or more computers Divide a target image containing an object and multiple character strings into multiple small regions; performing character recognition processing using the target image to recognize the plurality of character strings and recognize the positions of the plurality of character strings in the target image; assigning an index to each of the plurality of small regions corresponding to a relative positional relationship of the plurality of small regions in the target image; generating input data including input features in which position features obtained by encoding the index of the small region corresponding to a position of the character string are added to word features extracted from each of the plurality of character strings; The input data is input to a language model to obtain output data.

[0011] In one aspect of the present disclosure, the program On one or more computers, Divide a target image containing an object and multiple character strings into multiple small regions; performing character recognition processing using the target image to recognize the plurality of character strings and recognize the positions of the plurality of character strings in the target image; assigning an index to each of the plurality of small regions corresponding to a relative positional relationship of the plurality of small regions in the target image; generating input data including input features in which position features obtained by encoding the index of the small region corresponding to a position of the character string are added to word features extracted from each of the plurality of character strings; The program is for executing the step of inputting the input data into a language model and obtaining output data. [Effects of the Invention]

[0012] According to an example of the present disclosure, it is possible to accurately identify more character strings related to a target object from a target image. [Brief explanation of the drawings]

[0013] [Figure 1]1 is a block diagram showing an outline of the configuration of a first information processing device according to the present disclosure. [Figure 2] 10 is a flowchart showing an outline of a processing operation of the first information processing device according to the present disclosure. [Figure 3] FIG. 2 is a block diagram showing a detailed example of the configuration of a first information processing device according to the present disclosure. [Figure 4] 10 is a flowchart showing a detailed example of the processing operation of the first information processing device according to the present disclosure. [Figure 5] FIG. 2 is a block diagram showing an example configuration of a first input data generation unit according to the present disclosure. [Figure 6] 10 is a flowchart showing an example of a processing operation of a first input data generating unit according to the present disclosure. [Figure 7] FIG. 1 is a diagram illustrating an example of a target image according to the present disclosure. [Figure 8] FIG. 1 is a diagram illustrating a first example of a method for dividing a target image into a plurality of small regions according to the present disclosure. [Figure 9] FIG. 10 is a diagram showing an example of indexes assigned to each of a plurality of small regions obtained by dividing a target image according to the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of input data according to the present disclosure. [Figure 11] FIG. 2 is a diagram illustrating an example of the physical configuration of a first information processing device according to the present disclosure. [Figure 12] FIG. 10 is a diagram illustrating a second example of a method for dividing a target image into a plurality of small regions according to the present disclosure. [Figure 13] FIG. 10 is a diagram illustrating a third example of a method for dividing a target image into a plurality of small regions according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0015] <Embodiment 1> (overview) As shown in FIG. 1, the first information processing apparatus 100 includes a dividing unit 140, a recognizing unit 150, an indexing unit 160, an input data generating unit 170, and an output data obtaining unit 180.

[0016] The dividing unit 140 divides a target image including an object and a plurality of character strings into a plurality of small regions.

[0017] The recognition unit 150 performs character recognition processing using the target image to recognize a plurality of character strings and recognize the positions of the plurality of character strings in the target image.

[0018] The index assigning unit 160 assigns an index to each of the plurality of small regions in the target image, the index being associated with the relative positional relationship between the plurality of small regions.

[0019] The input data generation unit 170 generates input data including input features in which position features obtained by encoding the index of a small region corresponding to the position of each of a plurality of character strings are added to word features extracted from the character strings.

[0020] The output data acquisition unit 180 acquires output data by inputting input data into a language model.

[0021] According to this information processing device 100, it is possible to obtain output data processed by a language model for multiple character strings contained in an object and its surroundings in a target image, taking into account the relative positional relationships of the multiple character strings. By using such output data, it is possible to identify character strings related to the object from not only the character strings shown on the object but also a wide range of character strings contained in areas other than the object in the target image. Therefore, it is possible to accurately identify more character strings related to the object from the target image.

[0022] The first information processing apparatus 100 executes a first information process as shown in the flowchart of FIG.

[0023] The dividing unit 140 divides a target image including an object and a plurality of character strings into a plurality of small regions (step S140).

[0024] The recognition unit 150 performs character recognition processing using the target image to recognize a plurality of character strings and recognize the positions of the plurality of character strings in the target image (step S150).

[0025] The index assigning unit 160 assigns an index to each of the plurality of small regions in the target image, the index being associated with the relative positional relationship between the plurality of small regions (step S160).

[0026] The input data generation unit 170 generates input data including input features in which position features that encode the index of a small region corresponding to the position of each of a plurality of character strings are added to word features extracted from the character strings (step S170).

[0027] The output data acquisition unit 180 acquires output data obtained by inputting the input data into a language model (step S180).

[0028] According to this information processing method, it is possible to obtain output data processed by a language model for multiple character strings contained in an object and its surroundings in a target image, taking into account the relative positional relationships of the multiple character strings. By using such output data, it is possible to identify character strings related to the object from not only the character strings shown on the object but also a wide range of character strings contained in areas other than the object in the target image. Therefore, it is possible to accurately identify more character strings related to the object from the target image.

[0029] A detailed example of the first information processing device 100 will be described below.

[0030] (Detailed example) The first information processing device 100 according to the present disclosure includes, for example, as shown in FIG. 3, a target image storage unit 110, an object detection unit 120, a target identification means 130, a division unit 140, a recognition unit 150, an index assignment unit 160, an input data generation unit 170, an output data acquisition unit 180, a related information acquisition unit 190, an output control unit 200, and an output unit 210.

[0031] The first information processing device 100 also executes a first information processing as shown in Fig. 4. The first information processing is started, for example, when a target image to be processed is identified from target images stored in a target image storage unit 110, which will be described in detail later, for example, in accordance with an instruction from a user. Note that the trigger for starting the first information processing is not limited to the example given here.

[0032] The target image storage unit 110 stores a target image, which is an image including at least one object and a plurality of character strings.

[0033] The object detection unit 120 acquires object information including the position of the object detected from the target image. (Step S120).

[0034] The object specifying means 130 specifies an object to be processed from among the detected objects (step S130).

[0035] The dividing unit 140 divides the target image into a plurality of small regions (step S140).

[0036] The recognition unit 150 performs character recognition processing using the target image (step S150). The recognition unit 150 recognizes a plurality of character strings included in the target image and recognizes the positions of the plurality of character strings in the target image.

[0037] The index assigning unit 160 assigns an index to each of the plurality of small regions in the target image, the index being associated with the relative positional relationship between the plurality of small regions (step S160).

[0038] The input data generation unit 170 generates input data including input features in which position features that encode the index of a small region corresponding to the position of each of a plurality of character strings are added to word features extracted from the character strings (step S170).

[0039] 5, the input data generation unit 170 includes a word feature acquisition unit 171, an encoding unit 172, and an adding unit 173. The input data generation unit 170 then executes an input data generation process (step S170) as shown in FIG.

[0040] The word feature acquiring unit 171 acquires a plurality of word feature amounts by inputting each of a plurality of character strings into a word feature extraction model (step S171).

[0041] The encoding unit 172 acquires position features obtained by encoding the indexes assigned to the plurality of small regions (step S172).

[0042] The adding unit 173 generates input data including a plurality of input features in which the position feature of the corresponding small region is added to each of a plurality of word features (step S173).

[0043] Referring again to Figures 3 and 4. The output data acquisition unit 180 acquires output data in which, for example, input features are input as tokens to a language model (step S180).

[0044] The related information acquisition unit 190 acquires related strings of the object obtained by inputting information for identifying the object and the output data into a related string extraction model (step S190). The related string extraction model is, for example, a machine learning model that has been trained to extract related strings related to an object included in an image from multiple strings included in the image.

[0045] The output control unit 200 causes the output unit 210 to output the object-related information (step S200).

[0046] Next, a more detailed example of the processing executed by the functional configurations 110 to 210 of the first information processing device 100 will be described.

[0047] (Regarding the target image storage unit 110) The target image storage unit 110 may store, for example, a target image taken by a photographing device such as a camera (not shown), and the target image storage unit 110 may store the target image taken by the photographing device in advance.

[0048] (Regarding the object detection unit 120) The object detection unit 120 acquires object information including the position of an object detected using, for example, an object detection model.

[0049] The object detection model is, for example, a machine learning model that has been trained to detect an object contained in an image from the image. The machine learning model is constructed using, for example, a neural network, and the same applies hereinafter.

[0050] For example, when an image is input to an object detection model, the object detection model detects an object contained in the image and outputs object information including the position of the object in the image.

[0051] The position of an object is, for example, the position of a predetermined point or region related to the object. For example, the position of an object may be the position of the center of gravity of the region occupied by the object in the image. For example, the position of an object may be the position of the region indicated by a frame of a predetermined shape (e.g., a rectangle) surrounding the object in the image. Furthermore, if the image includes multiple objects, the object detection model may output object information including the position of each object. The object information may further include object identification information for identifying one or more detected objects.

[0052] The object detection model may be constructed, for example, by performing supervised learning using training data including learning images and objects and their positions included in the learning images. However, the learning method for the object detection model is not limited to this.

[0053] For example, the object detection unit 120 acquires a target image designated by the user from the target image storage unit 110. Note that the method by which the object detection unit 120 acquires the target image is not limited to the method of acquiring it from the target image storage unit 110. For example, the object detection unit 120 may acquire the target image from a photographing device that photographs the target image, an external device that stores the target image, or the like via a communication network. The communication network is, for example, a wired network, a wireless network, or a network configured by combining these, and the same applies hereinafter.

[0054] FIG. 7 is a diagram showing an example of a target image. The target image shown in the figure includes at least one product, a product shelf on which the product is placed, and a product tag attached to the product shelf. The product is an example of an object included in the target image. The product tag is a tag on which a string of characters related to the product is written, and is attached, for example, to a product shelf. The product tag may include, for example, one or more of the product's associated product name, price, product features, and promotional text. The target image may include one or more product tags. The product tag may include one or more strings of characters. Therefore, the target image shown in the figure includes at least one product as an object, and multiple strings of characters ST1 to ST9 written on at least one product tag.

[0055] The character string may be any character string that is included in the target image, and is not limited to a character string written on a product tag. For example, the character string may be a character string written on a product package or the like.

[0056] For example, the object detection unit 120 may include an object detection model, and the acquired target image may be input to the object detection model. Object information may be output from the object detection model. As a result, the object detection unit 120 may acquire object information including the position of the object detected from the target image.

[0057] The object detection model may be provided in an information processing device (not shown) that is provided outside the first information processing device 100 and connected to the first information processing device 100 via a communication network for transmitting and receiving information to and from the first information processing device 100. For example, the first information processing device 100 may transmit a target image to the external information processing device. Then, the external information processing device may input the target image to the object detection model, generate object information as a result, and transmit the generated object information to the first information processing device 100. This also allows the object detection unit 120 to acquire the object information.

[0058] For example, the object information may be information that associates, for each object (for example, a product) included in the target image, object identification information of the object with the position of the object in the image.

[0059] (Regarding the target identification means 130) The object identification means 130 identifies an object from among the detected objects, for example, based on a user's designation. The object is an object (e.g., a product) that is the target of processing. For example, the user may designate the object from within the target image displayed on the output unit 210. For example, the object may be designated by designating an area occupied by the object in the displayed target image. For example, an image that associates the target image with object identification information in the target image may be displayed, and the object may be designated by designating the object identification information. These methods allow the user to designate, as the object, an object (e.g., a product) in the target image for which related strings are to be automatically extracted. In this case, the user may designate, as the object, a product that may contain strings related to the product in an area other than the product's packaging. In FIG. 7, objects within the product included in the target image are indicated by hatching.

[0060] The method for specifying the object is not limited to the example given here.

[0061] In the following, an example in which there is one object will be described, but there may be multiple objects. In this case, the processing described below may be performed for each object. That is, for example, steps S140 to S190 may be performed for each object, and the results may be output together (step S200).

[0062] (Regarding the division unit 140) The dividing unit 140 divides the target image into a plurality of small regions according to, for example, a predetermined division rule. The division rule may be determined as appropriate.

[0063] For example, the division rule may include a rule for dividing the target image into a plurality of small regions using the position of the object in the target image as a reference position. That is, the division unit 140 may divide the target image into a plurality of small regions using the position of the object in the target image as a reference position. This reduces the possibility that the position of the object will be on the boundary of the small regions or in a biased position near the boundary. Therefore, it becomes possible to accurately identify the position of the object in the target image using the small regions.

[0064] For example, the division rule may include a rule for dividing the target image into a plurality of small regions such that the area of ​​the small region becomes smaller the closer it is to the reference position. In other words, the area of ​​the plurality of small regions may become smaller the closer it is to the reference position. This allows the position of the target object to be specified more precisely using the small regions. Therefore, it becomes possible to specify the position of the target object in the target image with high accuracy using the small regions.

[0065] Figure 8 shows an example of dividing a target image into small regions. This figure shows an example in which the target image is divided into a rectangular matrix of small regions whose boundaries are indicated by dotted lines. The figure also shows an example in which the position of the target object in the target image is used as a reference position, and the area of ​​the small regions becomes smaller as they are closer to the reference position.

[0066] The small area (i, j) shown in the figure can be expressed by the following equations (1) and (2), for example, where the vertical position is x and the horizontal position is y.

[0067]

number

[0068]

number

[0069] In equation (1), x represents the vertical position of the small region in a predetermined coordinate system for the target image. i is an integer indicating the vertical position of the small region. In the example of FIG. 8, i is an integer between -3 and 3, with the small region corresponding to the target object being 0.

[0070] In equation (2), y represents the horizontal position of the small region in a coordinate system predetermined for the target image. j is an integer indicating the left-right position of the small region. In the example of FIG. 8, j is an integer between -4 and 4, with the small region corresponding to the target object being set to 0.

[0071] N and M are the number of divisions in the vertical and horizontal directions, respectively. In the example in the same figure, N is 7 and M is 9. K is a constant according to the range of values ​​of x and y. Also, || is the Gauss symbol, and |x| and |y| represent the maximum integer values ​​that do not exceed x and y, respectively.

[0072] The method for dividing the target image into small regions is not limited to the example given here. For example, the number of small regions, the shape of the small regions, etc. may be changed as appropriate. The small regions may have a predetermined shape (e.g., rectangle, square, etc.) of the same size.

[0073] (Regarding the recognition unit 150) As described above, the recognition unit 150 performs character recognition processing using the target image to recognize multiple character strings included in the target image and recognize the positions of the multiple character strings in the target image. The positions of the character strings are, for example, the center positions of the character strings. Note that the positions of the character strings are not limited to the center positions of the character strings, and may be a point other than the center that is predetermined in relation to the positions of the character strings (for example, the upper left corner).

[0074] Character recognition processing is processing for recognizing characters contained in an image, and may be performed using, for example, a technique used in general OCR (Optical Character Recognition).

[0075] For example, the character recognition process may be performed using a character recognition model. The character recognition model is, for example, a machine learning model that has been trained to recognize multiple character strings included in a target image and the positions of the multiple character strings in the target image. For example, when an image is input, the character recognition model recognizes character strings included in the image and outputs recognition result information including the positions of the character strings in the image.

[0076] The object detection model may be constructed, for example, by performing supervised learning using training data including a learning image and a plurality of character strings contained in the learning image and their positions. However, the learning method for the object detection model is not limited to this.

[0077] The recognition unit 150 may include a character recognition model, and input the target image to the character recognition model. Recognition result information is output from the character recognition model. This allows the recognition unit 150 to recognize multiple character strings included in the target image and to recognize the positions of the multiple character strings in the target image from the target image.

[0078] The character recognition model may be provided in an information processing device (not shown) that is provided outside the first information processing device 100 and connected to the first information processing device 100 via a communication network for transmitting and receiving information to and from the first information processing device 100. For example, the first information processing device 100 may transmit a target image to the external information processing device. The external information processing device may then input the target image to the character recognition model, generate recognition result information as a result, and transmit the generated information to the first information processing device 100. This also enables the recognition unit 150 to recognize multiple character strings included in the target image and recognize the positions of the multiple character strings in the target image from the target image.

[0079] (Regarding the index assignment unit 160) The index assigning unit 160 assigns an index to each of the multiple small regions, for example, according to a predetermined assignment rule. The index is an indicator for identifying each small region and can be associated with the relative positional relationship of the multiple small regions in the target image. In the following, an example will be described in which the index is a numerical value, but the index is not limited to a numerical value and may be a combination of appropriate letters, symbols, numerical values, etc.

[0080] Furthermore, the assignment rules may be determined as appropriate.

[0081] For example, the assignment rule may include a rule for assigning numbers to the small regions in a predetermined order that increases by 1. The predetermined order may be the order in which the small regions are arranged.

[0082] FIG. 9 is a diagram showing an example of indexes assigned to the small regions shown in FIG.

[0083] The figure shows an example in which numbers are assigned as indexes, increasing by one in value according to the arrangement of the small areas, starting from the top left small area and proceeding to the right, and then gradually proceeding downward.

[0084] If the index is IDX, the index IDX may be a value obtained by, for example, the formula IDX=M×i+j+L, where L is a constant for making the index IDX an integer equal to or greater than 0. In the example of FIG. 9, L is 31, and IDX is an integer between 0 and 62.

[0085] In the figure, to show the correspondence between the small regions and the objects and character strings ST1 to ST9, the objects and character strings ST1 to ST9 are particularly picked out and shown from the products, product shelves, character strings ST1 to ST9, etc. included in the target image. In the example of the figure, the object is shown in the small region with an IDX of 0. In the example of the figure, the character strings ST1 to ST9 are shown in the small regions with IDX of 46, 48, 37, 50, 60, 11, 18, 22, and 24, respectively.

[0086] The predetermined order is not limited to this, and may be an order following a predetermined direction (for example, clockwise, counterclockwise, etc.) in order from the small area closest to the reference position. Also, the assignment rules are not limited to those exemplified here.

[0087] (Regarding the input data generation unit 170) As described above, the input data generation unit 170 includes, for example, a word feature acquisition unit 171, an encoding unit 172, and an adding unit 173. As a result, the input data generation unit 170 generates input data including input features in which position features obtained by encoding the index of a small region corresponding to the position of each of a plurality of character strings are added to word features extracted from the character strings.

[0088] About the word feature acquisition unit 171 The word feature acquiring unit 171 acquires a plurality of word feature amounts by inputting each of a plurality of character strings into a word feature extraction model, for example, as described above.

[0089] The word feature extraction model is, for example, a machine learning model that has been trained to extract word features (word features). For example, when one or more words are input, the word feature extraction model outputs vectors representing each of the one or more words as word features. This allows words to be represented as points in a vector space. In this case, the word features may be word feature vectors represented by vectors. For example, word2vec may be used as such a technology.

[0090] The word feature extraction model may be a model that outputs a fixed-length vector representing a word, for example, when using a training sentence to learn a task such as predicting surrounding words from words contained in the sentence, or predicting words contained in the sentence from surrounding words.

[0091] The word feature acquisition unit 171 may include a word feature extraction model, and input a plurality of character strings recognized from a target image to the word feature extraction model. A word feature amount (e.g., a word feature vector) of each character string is output from the word feature extraction model. This allows the word feature acquisition unit 171 to acquire a plurality of word features by inputting each of a plurality of character strings to the word feature extraction model.

[0092] The word feature extraction model may be provided in an information processing device (not shown) that is provided outside the first information processing device 100 and connected to the first information processing device 100 via a communication network for transmitting and receiving information to and from the first information processing device 100. For example, the first information processing device 100 may transmit a plurality of character strings recognized from a target image to the external information processing device. The external information processing device may then input the plurality of character strings into word feature extraction, generate word features as the result, and transmit the generated word features to the first information processing device 100. This also allows the word feature acquisition unit 171 to acquire a plurality of word features obtained by inputting each of a plurality of character strings into the word feature extraction model.

[0093] Regarding the encoding unit 172 The encoding unit 172 acquires position feature amounts obtained by encoding the indexes assigned to each of the plurality of small regions, for example, as described above.

[0094] For example, a technique used in general Transformer position encoding may be applied to the index encoding process. That is, for example, the encoding unit 172 may perform encoding process on the index assigned to each of the multiple small regions to obtain position feature amounts in which each index is represented by a vector.

[0095] In more detail, for example, the position feature amount is represented by a d-dimensional vector. In this case, the value of the 2i-th component and the value of the 2i+1-th component of the position feature amount are expressed as follows: PE(pos, 2i)=sin(pos / 10000 2i / d ) and PE(pos,2i+1)=cos(pos / 10000 2i / d ) where pos represents a position and may be, for example, an index. Note that the formula for obtaining the position feature (i.e., the method for encoding the index) is not limited to this.

[0096] The encoding process may be performed by an information processing device (not shown) that is provided outside the first information processing device 100 and is connected to the first information processing device 100 via a communication network for transmitting and receiving information to and from the first information processing device 100. For example, the first information processing device 100 may transmit indexes assigned to each of the plurality of small regions to the external information processing device. The external information processing device may then perform encoding process on each index, generate position feature values ​​as the result, and transmit the generated position feature values ​​to the first information processing device 100. This also allows the encoding unit 172 to acquire position feature values ​​obtained by encoding the indexes assigned to each of the plurality of small regions.

[0097] About the additional section 173 The adding unit 173 generates input data including a plurality of input features by adding the position feature of the corresponding small region to each of a plurality of word features, for example, as described above.

[0098] The input data includes, for example, input features corresponding to each of a plurality of character strings. Each input feature is, for example, a feature obtained by adding a position feature, which encodes an index of a small region corresponding to the position of the character string, to a word feature extracted from the corresponding character string. Each input feature may be, for example, a vector (input feature vector) obtained by adding a position feature vector to the word feature vector of the corresponding character string.

[0099] FIG. 10 is a diagram showing an example of the configuration of input data. The example shows input data including tokens in which input features (input feature vectors) corresponding to each of a plurality of character strings are arranged in a line in ascending order of the values ​​indicated by the indexes of small regions. The dotted arrows in the figure indicate the input features that follow from an input feature. It should be noted that the input data only needs to maintain the relative positional relationships between the plurality of character strings, and the input features may be arranged in a predetermined order, not limited to ascending order of the values ​​indicated by the indexes of small regions, such as descending order of the values ​​indicated by the indexes.

[0100] For example, suppose that character strings ST1 to ST9 are displayed in small regions with IDXs of 46, 48, 37, 50, 60, 11, 18, 22, and 24, respectively. In this case, the input feature corresponding to the small region with IDX of 46 includes the word feature (word feature vector) of character string ST1 and a position feature (position feature vector) obtained by encoding 46. The input feature corresponding to the small regions with IDXs of 48, 37, 50, 60, 11, 18, 22, and 24 includes the word feature (word feature vector) of character strings ST1 to ST9 and a position feature (position feature vector) obtained by encoding the IDX value. In the same figure, detailed configuration examples are shown for input feature corresponding to the small regions with IDXs of 0, 46, 47, 48, 60, and 62, and detailed configuration examples corresponding to the other small regions are not shown.

[0101] For example, a predetermined value corresponding to the absence of a character string, that is, a blank, may be set to the word feature amount (word feature vector) of a small region that does not contain a character string.

[0102] Furthermore, for example, when a single small region contains multiple character strings, the input data may include input features for each character string. In this case, the multiple input features corresponding to each of the multiple character strings in a single small region may include position features (position feature vectors) that encode the IDX values ​​of the same small region. These multiple input features may be included in the input data in a predetermined order related to the position of the character string from which the word features (word feature vectors) included in each feature are extracted. This predetermined order may be determined as appropriate. For example, the predetermined order may be the order in which the character string positions are closest to a predetermined point in the small region (e.g., the top left point, the center, etc.) or a reference position in the target image.

[0103] By generating such input data, it is possible to generate data (token string) that associates word feature vectors (word feature vectors) for multiple character strings contained in the target image with their relative positional relationships and arranges them in a row.

[0104] Generally, character strings related to an object are often contained near the object's location. Therefore, if the input data retains the relative positional relationships of multiple character strings, it is possible to accurately identify more character strings related to the object using the input data.

[0105] Furthermore, for example, by dividing the target image into a plurality of small regions using the position of the target object as the reference position, it is possible to generate a token string that accurately indicates the relative positional relationship between the target object and each character string.

[0106] Furthermore, for example, by dividing the image into smaller regions with smaller areas closer to the object, it is possible to generate a token string that precisely indicates the relative positional relationship between the object and each character string.

[0107] Furthermore, for example, by combining these and using the position of the object as the reference position, and dividing it into small areas with smaller areas the closer it is to the object, it is possible to generate a token string that accurately and precisely indicates the relative positional relationship between the object and each character string.

[0108] (Regarding the output data acquisition unit 180) The output data acquisition unit 180 acquires output data by inputting input data as tokens to a language model, for example. The language model is, for example, a machine learning model that has been trained to perform language processing, and may be, for example, a large-scale language model (LLM). The output data includes, for example, features (output features) that indicate the relative positional relationship between each of a plurality of character strings included in the target image and the target object. The output features may be, for example, output vectors expressed as vectors.

[0109] The language model may be provided in an information processing device (not shown) that is provided outside the first information processing device 100 and connected to the first information processing device 100 via a communication network for transmitting and receiving information to and from the first information processing device 100. For example, the first information processing device 100 may transmit input data to the external information processing device. The external information processing device may then input the input data to the language model, generate output data as a result, and transmit the output data to the first information processing device 100. This also allows the output data acquisition unit 180 to acquire output data in which the input data is input to the language model as tokens.

[0110] (Regarding the related information acquisition unit 190) The related information acquisition unit 190 acquires related strings of the object obtained by inputting information for identifying the object and output data into a related string extraction model.

[0111] The related string extraction model is, for example, a machine learning model that has been trained to extract related strings related to an object included in an image from among multiple strings included in the image. For example, when information for identifying an object and output data are input, the related string extraction model outputs object-related information including related strings related to the object.

[0112] The information for identifying the target object may be, for example, information capable of identifying the position of the target object in the target image. The position of the target object here may be represented by a coordinate position in the target image or by an index of a small region. For example, the information for identifying the target object may include object information of the target object. For example, the information for identifying the target object may include an index of a small region corresponding to the position of the target object.

[0113] This allows the object-related information to be generated based on the relative positional relationship between each of the plurality of small regions in the target image and the object. The object-related information may include object information of the object.

[0114] The related string extraction model may be provided in an information processing device (not shown) that is provided outside the first information processing device 100 and connected to the first information processing device 100 via a communication network for transmitting and receiving information to each other. For example, the first information processing device 100 may transmit information for identifying an object and output data to the external information processing device. Then, the external information processing device may input the information for identifying an object and the output data to the related string extraction model, generate object-related information as a result, and transmit it to the first information processing device 100. This also enables the related information acquisition unit 190 to acquire related strings of an object.

[0115] (Regarding the output control unit 200 and the output unit 210) The output control unit 200 causes the output unit 210 to output various types of information. For example, the output control unit 200 causes the output unit 210 to output object-related information. The output method is typically display. In this case, the output unit 210 is, for example, a display, and the output control unit 200 causes the output unit 210 to display the various types of information.

[0116] The output method is not limited to display, and may be, for example, transmission of information. The output control unit 200 may cause the output unit 210 as a transmission unit to transmit various information such as object-related information to an external device. The destination device may be, for example, predetermined or may be specified by the user.

[0117] (Example of physical configuration of information processing device 100) The information processing device 100 physically includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070, as shown in FIG.

[0118] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and other components to each other is not limited to bus connection.

[0119] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0120] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0121] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing the functions of the device that includes it. The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing the function corresponding to that program module.

[0122] The network interface 1050 is an interface for connecting a device equipped with the network interface 1050 to a communication network.

[0123] The input interface 1060 is an interface for the user to input information, and is configured from, for example, a touch panel, a keyboard, a mouse, and the like.

[0124] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0125] In this way, the functions of the information processing device 100 can be realized by the physical components cooperating to execute a software program. Therefore, the present invention may be realized as a software program or as a storage medium on which the program is non-temporarily recorded. Note that the information processing device may be physically composed of multiple devices (e.g., computers, etc.).

[0126] (Actions and Effects) As described above, according to this embodiment, the information processing device 100 includes the dividing unit 140, the recognizing unit 150, the indexing unit 160, the input data generating unit 170, and the output data obtaining unit 180.

[0127] The dividing unit 140 divides a target image containing an object and multiple character strings into multiple small regions. The recognition unit 150 performs character recognition processing using the target image to recognize multiple character strings and recognize the positions of the multiple character strings in the target image. The index assignment unit 160 assigns an index to each of the multiple small regions, which corresponds to the relative positional relationship of the multiple small regions in the target image. The input data generation unit 170 generates input data including input features in which word features extracted from each of the multiple character strings are added with position features that encode the index of the small region corresponding to the position of the character string. The output data acquisition unit 180 acquires output data by inputting the input data to a language model.

[0128] This makes it possible to obtain output data processed by a language model for multiple character strings contained in the target object and its surroundings in the target image, taking into account the relative positional relationships of the multiple character strings. By using such output data, it is possible to identify character strings related to the target object from not only the character strings shown on the target object but also a wide range of character strings contained in areas other than the target object in the target image. Therefore, it is possible to accurately identify more character strings related to the target object from the target image.

[0129] According to this embodiment, the input data generation unit 170 includes a word feature acquisition unit 171, an encoding unit 172, and an adding unit 173. The word feature acquisition unit 171 acquires a plurality of word features by inputting each of a plurality of character strings into a word feature extraction model. The encoding unit 172 acquires a position feature by encoding an index assigned to each of a plurality of small regions. The adding unit 173 generates input data including a plurality of input features by adding the position feature of the corresponding small region to each of a plurality of word features.

[0130] This allows for generating input data that takes into account the relative positional relationships of multiple character strings contained in the target object and its surroundings in the target image, and for processing the input data using a language model.By using output data that is the processing result of the language model, it is possible to identify character strings related to the target object from not only the character strings shown on the target object, but also a wide range of character strings contained in areas other than the target object in the target image.As a result, it becomes possible to accurately identify more character strings related to the target object from the target image.

[0131] According to this embodiment, the dividing unit 140 divides the target image into a plurality of small regions, using the position of the target object in the target image as the reference position.

[0132] This reduces the possibility that the object will be located at the boundary of a small region or at a biased position near the boundary, making it possible to accurately identify the position of the object in the target image using the small region.

[0133] According to this embodiment, the areas of the plurality of small regions are smaller the closer they are to the reference position.

[0134] This allows the position of the object to be specified more precisely using the small region, and therefore the position of the object in the target image can be specified with high accuracy using the small region.

[0135] According to this embodiment, the information processing device 100 includes a related information acquisition unit 190. The related information acquisition unit 190 acquires related strings of an object obtained by inputting information for identifying the object and output data into a related string extraction model that has been trained to extract related strings related to an object included in an image from multiple strings included in the image.

[0136] This makes it possible to obtain not only character strings displayed on the object but also character strings related to the object from a wide range of character strings included in areas other than the object in the target image, thereby enabling more character strings related to the object to be identified with high accuracy from the target image.

[0137] According to this embodiment, the information processing device 100 includes an object detection unit 120 and a target identification means 130. The object detection unit 120 acquires object information including the positions of objects detected from a target image. The target identification means 130 identifies a target object, which is an object to be processed, from among the detected objects.

[0138] This makes it possible to identify any object among the objects included in the target image as the target object, thereby enabling more character strings related to the target object to be identified with high accuracy from the target image.

[0139] According to this embodiment, the target image includes at least one product, a product shelf on which the product is placed, and a product tag attached to the product shelf. The target is a product identified from the at least one product.

[0140] This makes it possible to use an image of a product shelf on which a product is placed as a target image and identify any character strings contained therein that are related to the product. Therefore, it becomes possible to accurately identify more character strings related to the product from the target image, which is an image of a product shelf on which a product is placed.

[0141] <Embodiment 2> In the first embodiment, an example in which small regions are divided in a matrix has been described, but the method of dividing the target image into a plurality of small regions is not limited to dividing in a matrix. Furthermore, the plurality of small regions may divide at least a portion of the target image. In this case, the partial region of the target image divided into the plurality of small regions may include, for example, the target object.

[0142] The plurality of small regions may include concentric circles or spiral lines as boundaries centered on the reference position. The plurality of small regions may be divided radially.

[0143] FIG. 12 shows an example of dividing a target image into multiple concentric arc-shaped divided regions centered on a reference position. In the figure, the boundaries of the small regions are indicated by dotted lines. The index IDX of each small region may be a value obtained by, for example, the formula IDX=M[log(r+1) / K / N]+[θ / M]. r and θ are coordinate values ​​in a polar coordinate system centered on the position of the target image. N and M are the number of divisions in the radial and circumferential directions, respectively.

[0144] FIG. 13 shows an example of dividing a target image into a plurality of concentric arc-shaped divided regions centered on a reference position. In the figure, the boundaries of the small regions are indicated by dotted lines. The index IDX of each small region is, for example, IDX=Margin n |r-aθ n |+[θ / M], θ n ={θ+2nπ}. r,θ are the coordinate values ​​of the polar coordinate system centered on the position of the target image. M is the number of divisions in the circumferential direction. n represents the value of n that minimizes the target expression. n |r-aθ n| is r-aθ n represents the value of n that minimizes the absolute value of a. a is a parameter (constant) related to the logarithmic spiral.

[0145] (Actions and Effects) As described above, according to this embodiment, the boundaries of the multiple small regions include concentric circles or spiral lines centered on the reference position.

[0146] This allows the target image to be easily divided into a plurality of small regions with smaller areas closer to the reference position by dividing the image radially from the reference position. This makes it easy to precisely identify the position of the target object using the small regions. Therefore, it becomes possible to easily identify the position of the target object in the target image with high accuracy using the small regions.

[0147] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0148] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected.

[0149] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. A dividing means for dividing a target image including an object and a plurality of character strings into a plurality of small regions; a recognition unit that performs character recognition processing using the target image to recognize the plurality of character strings and recognizes positions of the plurality of character strings in the target image; an index assigning means for assigning an index to each of the plurality of small regions, the index being associated with a relative positional relationship of the plurality of small regions in the target image; an input data generation means for generating input data including input features obtained by adding position features obtained by encoding the indexes of the small regions corresponding to positions of the character strings to word features extracted from each of the plurality of character strings; and an output data acquisition means for acquiring output data by inputting the input data into a language model. Information processing device. 2. The input data generating means a word feature acquisition means for acquiring the plurality of word features by inputting each of the plurality of character strings into a word feature extraction model; an encoding means for acquiring position feature amounts obtained by encoding the indexes assigned to the plurality of small regions; and adding means for generating input data including a plurality of input features by adding a position feature of the corresponding small region to each of the plurality of word features. 1. An information processing device according to claim 1. 3. The dividing means divides the target image into the plurality of small regions using a position of the target object in the target image as a reference position. 1. The information processing device according to 1. or 2. 4. The areas of the plurality of small regions are smaller as they are closer to the reference position. 3. An information processing device according to the present invention. 5. The plurality of small regions include concentric circles or spiral lines as boundaries centered on the reference position. 3. An information processing device according to the present invention. 6. The image processing device further includes a related information acquisition means for acquiring the related character string of the object obtained by inputting the information for identifying the object and the output data into a related character string extraction model that has been trained to extract related character strings related to an object included in an image from a plurality of character strings included in the image. 6. An information processing device according to any one of 1. to 5. 7. an object detection means for acquiring object information including a position of an object detected from the target image; and an object specifying unit for specifying, from among the detected objects, the object to be processed. 10. An information processing device according to any one of 1. to 6. 8. the target image includes at least one product, a product shelf on which the product is placed, and a product tag attached to the product shelf; The object is a product identified from the at least one product. 8. An information processing device according to any one of 1. to 7. 9. One or more computers Divide a target image containing an object and multiple character strings into multiple small regions; performing character recognition processing using the target image to recognize the plurality of character strings and recognize the positions of the plurality of character strings in the target image; assigning an index to each of the plurality of small regions corresponding to a relative positional relationship of the plurality of small regions in the target image; generating input data including input features in which position features obtained by encoding the index of the small region corresponding to a position of the character string are added to word features extracted from each of the plurality of character strings; The input data is input to a language model to obtain output data. Information processing methods. 10. In generating the input data, acquiring the plurality of word features by inputting each of the plurality of character strings into a word feature extraction model; acquiring position features obtained by encoding the indexes assigned to the plurality of small regions; generating input data including a plurality of input features obtained by adding the position feature of the corresponding small region to each of the plurality of word features; 9. The information processing method described in paragraph 9. 11. In the division into the plurality of small regions, the position of the object in the target image is used as a reference position to divide the target image into the plurality of small regions. The information processing method according to 9. or 10. 12. The areas of the plurality of small regions are smaller as they are closer to the reference position. 11. The information processing method described above. 13. The plurality of small regions include concentric circles or spiral lines as boundaries centered on the reference position. 11. The information processing method described above. 14. Furthermore, the information for identifying the object and the output data are input to a related string extraction model that has been trained to extract related strings related to an object included in an image from a plurality of strings included in the image, and the related strings of the object are obtained. 14. An information processing method according to any one of 9. to 13. 15. Furthermore, object information including the position of the object detected from the target image is acquired; Identifying the object to be processed from among the detected objects. An information processing method according to any one of items 9 to 14. 16. the target image includes at least one product, a product shelf on which the product is placed, and a product tag attached to the product shelf; The object is a product identified from the at least one product. An information processing method according to any one of items 9 to 15. 17. On one or more computers, Divide a target image containing an object and multiple character strings into multiple small regions; performing character recognition processing using the target image to recognize the plurality of character strings and recognize the positions of the plurality of character strings in the target image; assigning an index to each of the plurality of small regions corresponding to a relative positional relationship of the plurality of small regions in the target image; generating input data including input features in which position features obtained by encoding the index of the small region corresponding to a position of the character string are added to word features extracted from each of the plurality of character strings; and obtaining output data by inputting the input data into a language model. 18. In generating the input data, acquiring the plurality of word features by inputting each of the plurality of character strings into a word feature extraction model; acquiring position features obtained by encoding the indexes assigned to the plurality of small regions; generating input data including a plurality of input features obtained by adding the position feature of the corresponding small region to each of the plurality of word features; 17. The program described in. 19. In the division into the plurality of small regions, the position of the object in the target image is used as a reference position to divide the target image into the plurality of small regions. 17. or 18. The program according to claim 17. 20. The areas of the plurality of small regions are smaller as they are closer to the reference position. 19. The program described in. twenty one. The plurality of small regions include concentric circles or spiral lines as boundaries centered on the reference position. 19. The program described in. twenty two. Furthermore, the information for specifying the object and the output data are input to a related string extraction model that has been trained to extract related strings relating to an object included in an image from a plurality of strings included in the image, thereby obtaining the related strings of the object. A program according to any one of 17. to 21. twenty three. Furthermore, object information including the position of the object detected from the target image is acquired; and specifying the object to be processed from among the detected objects. A program according to any one of 17. to 22. twenty four. the target image includes at least one product, a product shelf on which the product is placed, and a product tag attached to the product shelf; The object is a product identified from the at least one product. A program according to any one of 17. to 23. twenty five. A recording medium on which the program described in any one of 17. to 24. is recorded. [Explanation of symbols]

[0150] 100 Information processing device 110 Target image storage unit 120 Object detection unit 130 Target Identification Methods 140 Split section 150 Recognition part 160 Indexing Unit 170 Input data generation unit 171 Word feature acquisition unit 172 Encoding section 173 Additional Section 180 Output data acquisition unit 190 Related Information Acquisition Department 200 Output control section 210 Output section

Claims

1. A dividing means for dividing a target image including an object and a plurality of character strings into a plurality of small regions; a recognition unit that performs character recognition processing using the target image to recognize the plurality of character strings and recognizes positions of the plurality of character strings in the target image; an index assigning means for assigning an index to each of the plurality of small regions, the index being associated with a relative positional relationship of the plurality of small regions in the target image; an input data generation means for generating input data including input features obtained by adding position features obtained by encoding the indexes of the small regions corresponding to positions of the character strings to word features extracted from each of the plurality of character strings; and an output data acquisition means for acquiring output data by inputting the input data into a language model. Information processing device.

2. The input data generating means a word feature acquisition means for acquiring the plurality of word features by inputting each of the plurality of character strings into a word feature extraction model; an encoding means for acquiring position feature amounts obtained by encoding the indexes assigned to the plurality of small regions; and adding means for generating input data including a plurality of input features by adding a position feature of the corresponding small region to each of the plurality of word features. The information processing device according to claim 1 .

3. The dividing means divides the target image into the plurality of small regions using a position of the target object in the target image as a reference position.

3. The information processing device according to claim 1 or 2.

4. The areas of the plurality of small regions are smaller as they are closer to the reference position. The information processing device according to claim 3 .

5. The plurality of small regions include concentric circles or spiral lines as boundaries centered on the reference position. The information processing device according to claim 3 .

6. The image processing device further includes a related information acquisition means for acquiring the related character string of the object obtained by inputting the information for identifying the object and the output data into a related character string extraction model that has been trained to extract related character strings related to an object included in an image from a plurality of character strings included in the image.

3. The information processing device according to claim 1 or 2.

7. an object detection means for acquiring object information including a position of an object detected from the target image; and an object specifying unit for specifying, from among the detected objects, the object to be processed.

3. The information processing device according to claim 1 or 2.

8. the target image includes at least one product, a product shelf on which the product is placed, and a product tag attached to the product shelf; The object is a product identified from the at least one product.

3. The information processing device according to claim 1 or 2.

9. One or more computers Divide a target image containing an object and multiple character strings into multiple small regions; performing character recognition processing using the target image to recognize the plurality of character strings and recognize the positions of the plurality of character strings in the target image; assigning an index to each of the plurality of small regions corresponding to a relative positional relationship of the plurality of small regions in the target image; generating input data including input features in which position features obtained by encoding the index of the small region corresponding to a position of the character string are added to word features extracted from each of the plurality of character strings; The input data is input to a language model to obtain output data. Information processing methods.

10. On one or more computers, Divide a target image containing an object and multiple character strings into multiple small regions; performing character recognition processing using the target image to recognize the plurality of character strings and recognize the positions of the plurality of character strings in the target image; assigning an index to each of the plurality of small regions corresponding to a relative positional relationship of the plurality of small regions in the target image; generating input data including input features in which position features obtained by encoding the index of the small region corresponding to a position of the character string are added to word features extracted from each of the plurality of character strings; and obtaining output data by inputting the input data into a language model.

Images