Information processing system, information processing method, and information processing program

Abstract

This invention provides an information processing system, an information processing method, and an information processing program that can achieve dynamic tagging according to the processing context and generate highly practical modality data. [Solution] The AI ​​agent of the information processing system comprises: acquisition means capable of obtaining feature information from modality data; context determination means capable of determining the processing context when assigning tags to the feature information; and annotation means capable of assigning different tags to the same feature information according to the processing context determined by the context determination means. The processing context includes information regarding the purpose of the processing to be performed after tagging the feature information.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program capable of assigning dynamic tags according to a processing context.

Background Art

[0002] Conventionally, there has been proposed a system including an AI assistant for an automatic tagging system for materials and products, which incorporates a unit that analyzes the content of images and texts to automatically generate related tags, a unit that preferentially assigns highly relevant tags by referring to user search history and purchase history, and a user feedback function for improving tag accuracy (for example, see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] However, although conventional systems can automate tagging by automatically generating tags, it has been difficult to assign appropriate tags depending on the content of the image or text to be tagged.

[0005] In view of such circumstances, the present invention intends to provide an information processing system, an information processing method, and an information processing program that can realize dynamic tagging according to a processing context and generate highly practical modality data.

Means for Solving the Problems

[0006] The information processing system according to the present invention is an information processing system configured to have a storage means and an AI agent, wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent comprises an acquisition means capable of obtaining feature information from the modality data, a context determination means capable of determining the processing context when assigning tags to the feature information, and an annotation means capable of assigning different tags to the same feature information according to the processing context determined by the context determination means. The processing context includes information regarding the purpose of the processing to be performed after tagging the feature information. This is an information processing system characterized by the following:

[0007] The information processing method according to the present invention is an information processing method performed by an information processing system configured to have a storage means and an AI agent, wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is capable of performing at least an acquisition step in which feature information can be obtained from the modality data, a context determination step in which the processing context for assigning tags to the feature information can be determined, and an annotation step in which different tags can be assigned to the same feature information according to the processing context determined by the context determination step. Furthermore, the processing context includes information regarding the purpose of the processing to be performed after tagging the feature information. This is an information processing method characterized by the following features.

[0008] The information processing program according to the present invention is a program for an information processing system comprising a storage means and an AI agent, wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent functions as an acquisition means capable of obtaining feature information from the modality data, a context determination means capable of determining the processing context when assigning tags to the feature information, and an annotation means capable of assigning different tags to the same feature information according to the processing context determined by the context determination means. The processing context includes information regarding the purpose of the processing to be performed after tagging the feature information. This is an information processing program characterized by the following: [Effects of the Invention]

[0009] The information processing system, information processing method, and information processing program according to the present invention can achieve dynamic tagging according to the processing context and generate highly practical modality data, which is an excellent effect. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing an overview of the information processing system 10 according to this embodiment 1. [Figure 2] This is a system configuration diagram showing an example of the configuration of the information processing system 10 according to this embodiment 1. [Figure 3] This is a schematic diagram showing an overview of the information processing system 30 related to the modified example. [Figure 4] This is a schematic diagram showing an overview of the information processing system 10 according to this second embodiment. [Figure 5] This is a system configuration diagram showing an example of the configuration of the information processing system 10 according to this second embodiment. [Figure 6] This is a schematic diagram showing an overview of the information processing system 10 according to this third embodiment. [Figure 7] This is a system configuration diagram showing an example of the configuration of the information processing system 10 according to this third embodiment. [Figure 8] This diagram shows an example of multiple types of tags that can be attached to a single purchase information item. [Modes for carrying out the invention]

[0011] <<Embodiment 1>> The information processing system 10 according to Embodiment 1 of the present invention will be described below with reference to the drawings.

[0012] <Overview of the Information Processing System> First, an overview of the information processing system 10 according to this embodiment 1 will be described using Figure 1. Figure 1 is a schematic diagram showing an overview of the information processing system 10 according to this embodiment 1.

[0013] The information processing system 10 according to the present embodiment is an information processing system configured to include a storage unit 14 and an AI agent 12, wherein the storage unit 14 is capable of storing multimodal data including a plurality of modality data related to the same target, and the AI agent 12 includes an acquisition unit 12a capable of acquiring feature information from the modality data, a collation unit 12b capable of collating a plurality of pieces of feature information acquired from at least two pieces of modality data and detecting inconsistency among the plurality of pieces of feature information, a replacement unit 12c capable of executing replacement processing that, when an inconsistency is detected, designates one of the plurality of pieces of feature information as correct feature information (positive feature information) and replaces feature information different from the positive feature information (erroneous feature information) with the positive feature information, and an annotation unit 12d capable of adding a tag to the modality data using the positive feature information. The information processing system is characterized by comprising the above components.

[0014] According to the information processing system 10 according to Embodiment 1 of the present embodiment, erroneous feature information can be corrected to correct feature information, and a tag can be added to the modality data based on the correct feature information, so that modality data with high reliability and high practicability can be generated.

[0015] Here, the "AI agent" refers to an AI system that performs inference, judgment, planning, information generation or process execution based on input information. The AI agent may be configured using at least one of machine learning models, deep learning models, large language models, generative AI, and other AI technologies, or may be configured by combining a plurality of AI technologies. The AI agent includes programs, software modules, or information processing apparatuses that implement these functions. The implementation form of the AI agent is not particularly limited, and for example, it may be implemented as Cloud AI, Fog AI, edge AI, or on-device AI (including embedded AI). In addition, the AI agent may be configured as a cloud type, a local type, a distributed type, or a combination thereof.

[0016] An AI agent executes processing based on an externally provided activation trigger (for example, a question (prompt) from a user, a signal input from a sensor, or reception of an e-mail), or executes processing based on an internally provided activation trigger (for example, activation by an internal scheduler, occurrence of an abnormality, or update of specific information).

[0017] For example, when a question (prompt) is given as an external activation trigger, tasks such as searching for an answer to the question or generating an answer are executed, and the answer to the question is output. Further, for example, when a problem (prompt) is given as an external activation trigger, tasks such as searching for a solution to the problem or generating a solution are executed, and the solution to the problem is output.

[0018] Further, the term "prompt" refers to an instruction or question that a user gives to a system, and includes, for example, an AI prompt given to generative AI, a command prompt that gives an instruction by a command statement, and the like. The data format of the "prompt" is not particularly limited, and may be, for example, any of text data, audio data, still image data, video data, files, and the like.

[0019] Various processes executed by the AI agent can also be executed by "agent-type AI", "optical operation model", "multimodal AI agent", or the like. Here, the "agent-type AI" refers to a system in which one or more AI agents autonomously perform inference, planning, decision-making, or process execution in order to achieve a predetermined purpose. When a plurality of AI agents are used in the agent-type AI, each AI agent takes charge of a specialized role, and can output an answer to a question, a solution to a problem, or the like while mutually exchanging information.

[0020] Furthermore, an "optical computing model" refers to a system that performs information processing using the physical properties of light. For example, by combining lenses, prisms, diffraction gratings, optical waveguides, and other optical elements, it is possible to perform calculations or information processing using optical signals. Because optical computing models can process a large number of optical signals in parallel, they can sometimes achieve faster processing or lower power consumption compared to systems that perform calculations using electrical signals.

[0021] Furthermore, a "multimodal AI agent" refers to an AI system that acquires information from multiple modalities, including text, images, audio, video, or sensor data, integrates this information to recognize the state of the environment or object, generates a plan to achieve an objective, and autonomously executes one or more processes according to that plan.

[0022] "Modality data" refers to data belonging to each information type, such as text data, image data, video data, audio data, and sensor data. Modality data is data that is tagged by annotation. For example, "text data" includes documents, emails, prompts, chat logs, papers, and program code. "Image data" includes photographs, drawings, blueprints, and graphs. "Video data" includes images captured by cameras, commercials, promotional videos, and movies. "Audio data" includes prompts, conversations, music, instrument sounds, and animal sounds. "Sensor data" includes data acquired by various sensors, such as temperature data, humidity data, acceleration data, angular velocity data, position data, pressure data, and heart rate data. "Multimodal data" refers to data that includes modality data of multiple different information types related to the same subject.

[0023] "Feature information" refers to information extracted from text information, numerical information, tag information, vector information, image features, audio features, and other modality data. Examples of "feature information" include words, keywords, sentences, summaries, tags, category information, sentiment information, and vector representations (embedded vectors) extracted from text data; object information, person information, region information, color features, and shape features extracted from image data; phonological information, speaker information, and sentiment information extracted from audio data; behavioral or event information extracted from video data; and state information, anomaly information, and prediction information extracted from sensor data.

[0024] "Tags" refer to semantic information that represents the content, attributes, meaning, state, classification, evaluation, or usage of the target data, and include labels, classification information, attribute information, metadata, identification information, category information, evaluation information, analysis result information, recommendation information, and any other information associated with the target data.

[0025] "Usage pattern" refers to the purpose of use, user, usage scenario, or method of use of the target data or the tags assigned to it. Examples of usage patterns include the following:

[0026] (Example 1) Differences in usage patterns by users When store staff refer to annotation information to optimize shelf layout, the same purchase information is tagged with "Recommended Display Item." On the other hand, when a buyer at headquarters refers to annotation information for purchasing planning, the same purchase information is tagged with "Order Priority: High." In this way, even with the same data, the tags assigned change dynamically depending on the user and purpose of use of the annotation information.

[0027] (Example 2) Differences in usage patterns depending on the usage scenario When a demand forecasting system references annotation information, the purchase information is tagged with "Increased winter demand." On the other hand, when a competitive analysis system references the same annotation information, the same purchase information is tagged with "High competitive advantage." In this way, even when different systems or applications are used, the context determination means can determine the usage pattern and dynamically switch the tags assigned.

[0028] "Annotation information" refers to information that associates one or more tags with the target data to which those tags are assigned.

[0029] <Example of system configuration> Next, an example of the configuration of the information processing system 10 according to this embodiment 1 will be described using Figure 2. Figure 2 is a system configuration diagram showing an example of the configuration of the information processing system 10 according to this embodiment 1.

[0030] The information processing system 10 can be configured, for example, to include a system terminal 12 that controls the entire system, and an information storage terminal 14 and an external terminal 16 that are connected to the system terminal 12 via a network NW so that they can communicate with each other.

[0031] The system terminal 12 is a terminal that controls the entire information processing system 10, and is composed of conventionally known servers, personal computers, etc. In this example, the system terminal 12 is composed of one server, but it may be composed of multiple servers, personal computers, etc. The hardware configuration of the system terminal 12 and the programs that the system terminal 12 executes will be described later.

[0032] The information storage terminal 14 is a terminal (storage means) for users (individuals or specific groups of multiple people (teams, departments, companies, etc.)) using the information processing system 10 to store multimodal data (annotation target data), and is composed of conventionally known servers, personal computers, etc.

[0033] In this example, the information storage terminal 14 is configured with a single server, but it may also be configured with multiple servers or personal computers. Furthermore, the terminal (storage means) for storing multimodal data may be an internal storage means (for example, the storage device 26 shown in Figure 2) connected to the system terminal 12 via a local network (for example, LAN or P2P), or it may be other storage means.

[0034] External terminals 16 are terminals used by users of the information processing system 10 (individuals or specific groups of people (teams, departments, companies, etc.)), and consist of personal computers, tablets, smartphones, etc. The type of external terminal 16 is not particularly limited, but examples include smartphones, personal computers, and tablets used by individuals, and smartphones, personal computers, and tablets used by employees of companies, etc.

[0035] The network NW is a line that allows the system terminal 12, the information storage terminal 14, and the external terminal 16 to communicate with each other, and is typically composed of a WAN (Wide Area Network), also known as the Internet. The network NW may be wired or wireless, a LAN (Local Area Network), a dedicated line such as a VPN (Virtual Private Network), or a combination of these lines.

[0036] <System Terminal / Hardware Configuration Example> Next, we will describe an example of the hardware configuration of system terminal 12.

[0037] As shown in Figure 2, the system terminal 12 is configured to include, for example, a CPU 21 and a ROM 22, RAM 23, external storage drive 25, storage device 26, input device 27, display device 28, communication unit 29, etc., all connected to the CPU 21 via a bus.

[0038] The CPU 21 is a control means that controls the entire system terminal 12, and performs processes such as executing application programs and operating systems (OS) stored in ROM 22 and storage devices 26, and storing data and files necessary for program execution in RAM 23 and storage devices 26.

[0039] ROM22 is a storage means for storing basic I / O programs and various data, and is composed of, for example, PROM, flash memory, etc. RAM23 is a storage means for temporarily storing data, and is composed of, for example, SDRAM, DRAM, etc. External storage drive25 is a control means that can read and write data to recording media 24 such as magnetic tape, DVD, etc., and is composed of, for example, magnetic tape storage, DVD drive, etc.

[0040] The storage device 26 is a storage means for storing application programs, the OS, control programs, related programs, various information, etc., and is composed of, for example, a hard disk drive (HDD), a solid-state drive (SDD), etc. The input device 27 is for inputting commands (instructions), etc., to the system terminal 12, and is composed of, for example, a keyboard, a pointing device (mouse, etc.), a touch panel, etc.

[0041] The display device 28 displays commands input by the input device 27, the response output of the system terminal 12 to those commands, and various other displays, and is composed of, for example, a liquid crystal display, plasma display, organic EL, etc. The communication unit 29 is a control means that controls communication with the information storage terminal 14 and external terminals 16, etc. via the network NW, and is composed of, for example, a communication card, etc.

[0042] <System Terminal / Function> Next, we will explain the functions of the system terminal 12.

[0043] The storage device 26 of the system terminal 12 stores a program (information processing program) that causes the system terminal 12 to function as an acquisition means 12a, a matching means 12b, a replacement means 12c, and an annotation means 12d.

[0044] <System terminal / Function / Acquisition method> Next, the acquisition means 12a will be described.

[0045] The acquisition means 12a is a means capable of acquiring characteristic information from modality data stored in the information storage terminal 14, etc., and in this example, it consists of a program stored in the storage device 26 of the system terminal 12, or the storage device 26, etc.

[0046] The acquisition means 12a, based on a specific activation trigger, refers to modality data stored in the information storage terminal 14, acquires characteristic information (character information, numerical information, tag information, vector information, image features, audio features, etc.) from the modality data, and stores the acquired characteristic information in the storage device 26 in association with the modality data.

[0047] The content of the "trigger" is not particularly limited and may be an externally provided trigger (e.g., a question (prompt) from a user of an external terminal 16, a signal input from an external sensor, or the receipt of an email from an external source) or an internally provided trigger (e.g., activation by an internal scheduler, occurrence of an anomaly, or update of specific information).

[0048] For example, if a user of an external terminal 16 stores a new document file (text data) in the information storage terminal 14, the acquisition means 12a, triggered by the fact that a new document file has been stored in the information storage terminal 14 (that specific information has been updated), acquires characteristic information (for example, sentences, strings, numbers, codes, etc. extracted from text data) from the document file using character recognition processing (OCR, etc.), and stores the characteristic information in the storage device 26 in association with the document file.

[0049] Furthermore, for example, if a user of the external terminal 16 provides a voice question (prompt) (if a user of the external terminal 16 inputs a question (prompt)), the acquisition means 12a, triggered by the provision of a voice question (prompt), acquires feature information (for example, phonological information extracted from the voice data, speaker information, emotion information, strings constituting the question, keywords of the question, etc.) from the voice data of the question (prompt) through speech recognition processing (ASR, etc.), and stores the feature information in the storage device 26 in association with the voice data.

[0050] Furthermore, for example, when the update date for updating modality data tags arrives (for example, once a week) (when triggered by the internal scheduler), the acquisition means 12a, triggered by the internal scheduler, acquires an arbitrary modality data from among the multiple modality data stored in the information storage terminal 14, for example, a photograph of a car (image data), and from the photograph of the car, it acquires feature information (for example, object information extracted from image data (car name, manufacturer name, model, etc.), person information, area information, color features, shape features, etc.) through image recognition processing, and Feature Information of, car The image is associated with the device and stored in the storage device 26.

[0051] <System terminal / Function / Verification method> Next, the verification means 12b will be described.

[0052] The matching means 12b is a means capable of matching multiple feature information obtained from at least two modality data and detecting inconsistencies in the multiple feature information. In this example, it is composed of a program stored in the storage device 26 of the system terminal 12, or the storage device 26, etc.

[0053] For example, consider a case where acquisition means 12a performs speech recognition (ASR) on the audio track of a promotional video stored in an information storage terminal 14, etc., and obtains audio-text information (text information extracted from audio data) as feature information from the audio track of the promotional video, which is "Collagen-containing beauty serum, regular price 3,280 yen". At the same time, optical character recognition (OCR) is performed on the video frames of the same promotional video, and obtains video-text information (text information extracted from video data) as feature information from the video frames of the promotional video, which is "Collagen-containing beauty serum, regular price 3,820 yen".

[0054] In this case, the matching means 12b compares the audio-text information and video-text information obtained from the two modality data and determines that there is an inconsistency (mismatch) in the characteristic information (product price information) of the two because the price notations of the two are different ("3,280 yen" and "3,820 yen"). Note that the matching is not limited to a comparison of exact string matches, but may also be performed for each semantic unit such as numbers, proper nouns, and product codes.

[0055] Furthermore, consider the case where, for example, acquisition means 12a performs OCR processing on image data of product packaging displayed on a drugstore shelf, and obtains image text information (text information extracted from image data) as feature information from the product packaging, such as "JAN:49012 Hyaluronic Acid Moisture Lotion," and also refers to the product master stored in the information storage terminal 14, and obtains master text information (text information extracted from product master text) as feature information from the product master, such as "JAN:49013 Hyaluronic Acid Moisture Lotion."

[0056] In this case, the matching means 12b compares the image character information obtained from the two modality data with the master character information and determines that there is an inconsistency (mismatch) in the characteristic information (product JAN code) of the two because their JAN codes ("49012" and "49013") are different.

[0057] Furthermore, consider a case where, for example, acquisition means 12a performs barcode recognition processing or OCR processing on image data of the store's shelf layout, obtains an image acquisition code "490120" (a JAN code extracted from the image data) as feature information from the shelf layout image data, refers to sales performance data from the ID-POS system, and obtains POS character information "490121" (a product code extracted from the sales performance data) corresponding to the same shelf area from the sales performance data.

[0058] In this case, the matching means 12b compares the image acquisition code obtained from the two modality data with the POS character information and determines that there is an inconsistency (mismatch) between the two characteristic information (codes assigned to the products) because the two codes ("490120" and "490121") are different.

[0059] The feature information that the matching means 12b matches may be feature information of different types. For example, the matching means 12b may match text information indicating a product name with category information indicating a product category. Specifically, if the text information indicating a product name is "potato chips" and the category information indicating a product category is "confectionery", the matching means 12b can determine that there is consistency between the text information and the category information. On the other hand, if the text information indicating a product name is "potato chips" but the category information indicating a product category is "beverage", the matching means 12b can determine that there is inconsistency between these feature pieces of information.

[0060] Furthermore, the matching means 12b may also compare the product type information recognized from the product image with the product name information obtained from the ID-POS system. For example, if "beverage" is recognized from the product image, but the product name information indicates confectionery, the matching means 12b can detect the inconsistency.

[0061] In this way, by comparing feature information of different types, it is possible to detect inconsistencies that would be difficult to detect when comparing feature information of the same type. Furthermore, since the target can be evaluated from multiple perspectives, the reliability and consistency of the feature information can be improved. Moreover, even if one set of feature information is missing or misrecognized, the other set of feature information can be used to detect the inconsistency, thereby improving the accuracy of information processing.

[0062] Furthermore, the feature information that the matching means 12b matches may be feature information obtained from modality data of different types.

[0063] For example, the matching means 12b may compare feature information obtained from image data with feature information obtained from text data. Specifically, if the product type information obtained from the product image by image recognition is "beverage" and the product type information obtained from the product description is "beverage," the matching means 12b can determine that these feature information are consistent. On the other hand, if the product type information obtained from the product image is "beverage," but the product type information obtained from the product description is "confectionery," the matching means 12b can determine that there is an inconsistency between these feature information.

[0064] Furthermore, the matching means 12b may also compare feature information obtained from voice data with feature information obtained from text data. For example, it may compare a product name obtained as a result of speech recognition of the user's utterance with a product name in the text entered by the user, and detect an inconsistency if the two do not match.

[0065] Furthermore, the matching means 12b may also compare the feature information obtained from the image data with the feature information obtained from the ID-POS system. For example, it may compare the product name recognized from the product image with the product name registered in the ID-POS system and detect an inconsistency if the two do not match.

[0066] In this way, by comparing feature information obtained from modality data of different types, inconsistencies, misrecognitions, input errors, or missing information can be detected with higher accuracy compared to using only a single modality data set. Furthermore, since information contained in multiple modality data can be complemented by each other, the reliability and consistency of feature information can be improved. In addition, even if some modality data contains noise or misrecognitions, correction or verification can be performed using feature information obtained from other modality data, thereby improving the accuracy and robustness of information processing.

[0067] <System terminal / Function / Replacement method> Next, the substitution means 12c will be described.

[0068] The replacement means 12c is a means capable of performing a replacement process in which, when an inconsistency in feature information is detected by the matching means 12b, one of the multiple feature information pieces is designated as the correct feature information (positive feature information), and feature information different from the correct feature information (incorrect feature information) is replaced (or modified, corrected, or corrected) with the correct feature information. In this example, it is composed of a program stored in the storage device 26 of the system terminal 12, or the storage device 26, etc.

[0069] For example, consider a case where acquisition means 12a acquires audio-text information (text information extracted from audio data) as feature information, "Collagen-containing beauty serum, regular price 3,280 yen," from the audio track of a promotional video stored in the information storage terminal 14, and also acquires video-text information (text information extracted from video data) as feature information, "Collagen-containing beauty serum, regular price 3,820 yen," from the video frames of the same promotional video, and matching means 12b determines that there is an inconsistency (mismatch) between the two sets of feature information (product price information).

[0070] In this case, the substitution means 12c, in accordance with a pre-set priority rule (in this example, video data is considered more important than audio data), treats the video text information as correct information, the audio text information that differs from the video text information as incorrect feature information, replaces the corresponding part of the audio text information ("3,280 yen") with the corresponding part of the video text information ("3,820 yen"), and stores the feature information "3,820 yen" (product price information) as correct feature information in the storage device 26.

[0071] Furthermore, the determination of which modality data is considered correct may be made dynamically based on factors such as the confidence score of the modality data, the data acquisition time, and the priority of the information sources. For example, if feature information obtained from a particular modality data source matches feature information obtained from other modality data sources at a high frequency, the confidence score of that modality data source may be increased. On the other hand, if feature information obtained from a particular modality data source contains many misrecognitions, inconsistencies, or correction histories, the confidence score of that modality data source may be decreased.

[0072] As demonstrated in this example, the system can learn the reliability of each modality data as it continues to operate. Compared to setting fixed priorities in advance, it can select appropriate modality data as positive feature information in response to environmental changes, changes in usage, or changes in the quality of information sources. As a result, the consistency and reliability of feature information can be improved, and the occurrence of replacement processes based on incorrect feature information can be suppressed.

[0073] Furthermore, for example, if the product name obtained from the first modality data is "Product A" and the product name obtained from the second modality data is "Product B," an inconsistency may be determined to exist between these feature pieces of information. In this case, the data acquisition time may be used as a priority rule for determining the correct feature information.

[0074] Specifically, if the acquisition time of the first modality data is more recent than the acquisition time of the second modality data, the feature information obtained from the first modality data may be determined as the true feature information. For example, if "Product B" is registered in the ID-POS system data acquired after a product name change, while "Product A" is obtained from image data acquired before the name change, the feature information based on the ID-POS system data with the more recent acquisition time can be determined as the true feature information.

[0075] As this example demonstrates, even when product information is updated, user information changes, or the state of the object changes, newer information can be prioritized. This suppresses inconsistencies caused by outdated information and improves the freshness and reliability of characteristic information.

[0076] Furthermore, for example, if the product name obtained from a product image is "Product A" and the product name obtained from an ID-POS system is "Product B," the matching means 12b may determine that there is an inconsistency between these feature pieces of information. In this case, the priority order of the information sources may be used as a priority rule for determining the correct feature information.

[0077] Specifically, if the ID-POS system, product master database, user input information, and image recognition results are set to have the highest reliability in a predetermined order, the replacement means 12c can determine the feature information obtained from the ID-POS system as correct feature information and the feature information obtained by image recognition as incorrect feature information. Furthermore, if feature information for the same product is obtained from the product master database, the ID-POS system, and product images, the feature information obtained from the product master database may be given the highest priority, followed by the feature information obtained from the ID-POS system.

[0078] As demonstrated in this example, since the correct feature information can be determined by considering the varying reliability of each information source, the impact of image recognition errors, input errors, or temporary data inconsistencies can be reduced. As a result, the consistency and reliability of feature information can be improved, and subsequent processing based on incorrect feature information can be suppressed.

[0079] Furthermore, the AI ​​agent may automatically update and learn the confidence score of modality data, the priority of information sources, etc., based on the performance and history of matching modality data. For example, in the process of repeatedly performing the matching process of modality data, the AI ​​agent may accumulate the performance of selecting feature information obtained from each modality data as positive feature information, and dynamically update the confidence score, the priority of information sources, etc., corresponding to each modality data based on the accumulated performance. In addition, the updated confidence score, the priority of information sources, etc., may be used when determining positive feature information in subsequent matching processes.

[0080] In this example, the AI ​​agent learns the reliability of each modality data and the reliability of the information source based on operational performance, and can preferentially select feature information obtained from more reliable modality data and information sources as positive feature information. For example, consider a case where acquisition means 12a obtains image text information (text information extracted from image data) as feature information, such as "JAN:49012 Hyaluronic Acid Moisture Lotion," from image data of a product package, and master text information (text information extracted from product master text) as feature information, such as "JAN:49013 Hyaluronic Acid Moisture Lotion," from the product master, and matching means 12b determines that there is an inconsistency (mismatch) between the two sets of feature information (product JAN codes).

[0081] In this case, the substitution means 12c, in accordance with a pre-set priority rule (in this example, the product master text is considered correct over the product image data), treats the master character information as correct information, the image character information that differs from the master character information as incorrect feature information, replaces the corresponding part of the image character information ("JAN:49012") with the corresponding part of the master character information ("JAN:49013"), and stores the feature information "JAN:49013" (the product's JAN code) as correct feature information in the storage device 26.

[0082] Furthermore, consider a case where, for example, acquisition means 12a acquires an image acquisition code "490120" (a JAN code extracted from image data) as feature information from shelf layout image data, and also acquires POS character information "490121" (a product code extracted from sales performance data) corresponding to the same shelf area from sales performance data of the ID-POS system, and matching means 12b determines that there is an inconsistency (mismatch) between the two sets of feature information (codes assigned to products).

[0083] In this case, the substitution means 12c, in accordance with a pre-set priority rule (in this example, the data from the ID-POS system is considered more important than the image data), treats the POS character information as correct information, treats the image acquisition code that differs from the POS character information as incorrect feature information, replaces the corresponding part of the image acquisition code ("490120") with the corresponding part of the POS character information ("490121"), and stores the feature information "490121" (the code assigned to the product) as correct feature information in the storage device 26.

[0084] Furthermore, the system may include a means for assigning an inconsistency score according to the degree of inconsistency, and the replacement means 12c may only replace the incorrect feature information with correct feature information if the inconsistency score exceeds a predetermined threshold.

[0085] Specifically, the assignment means includes the degree of agreement, similarity, frequency of occurrence, and semantic proximity between the incorrect feature information and the correct feature information, and teeth,The degree of inconsistency is calculated based on estimation results from a machine learning model, and an inconsistency score is assigned according to the degree of inconsistency. For example, if, according to a pre-set priority rule (for example, a rule that prioritizes information obtained from the ID-POS system over information obtained from image data), character information obtained from the ID-POS system is treated as positive feature information and character information obtained from image data is treated as negative feature information, the assignment means calculates the inconsistency score based on the string similarity, edit distance, pronunciation similarity, or distance between embedding vectors between the positive and negative feature information.

[0086] Furthermore, if the mismatch score is above a predetermined threshold (indicating a high degree of mismatch), the replacement means 12c automatically replaces the erroneous feature information with the corresponding correct feature information. If the mismatch score is below a predetermined threshold (indicating a low degree of mismatch), the replacement is not performed, and confirmation by the user may be requested. This makes it possible to achieve highly accurate automatic correction to correct feature information while suppressing the occurrence of incorrect replacements.

[0087] Furthermore, when the substitution means 12c selects positive feature information from multiple feature information, the "pre-set priority rules" include, for example, a rule that prioritizes information obtained from the ID-POS system over information obtained from image data, a rule that prioritizes input information by users of the information processing system over information obtained from other sources, a rule that prioritizes master data registered by the administrator, a rule that prioritizes information with a more recent acquisition date and time, a rule that prioritizes information with a high confidence score, a rule that prioritizes information obtained from a source with a pre-set priority among multiple sources, a rule that prioritizes information that appears frequently among information obtained multiple times, a rule that prioritizes recognition results with a high confidence score among recognition results from multiple recognition models, and a rule that prioritizes the majority vote result of information obtained from multiple sources. Furthermore, the matching means 12b can match multiple feature information obtained from at least three modality data and detect inconsistencies in the multiple feature information, and the replacement means 12c may, when an inconsistency is detected, be able to determine the correct feature information from the multiple feature information by majority vote and perform a replacement process to replace the incorrect feature information with the correct feature information.

[0088] For example, if, for the same product, the product name obtained from the image data of the product package is "Chocolate A", the product name obtained from the ID-POS system is "Chocolate A", and the product name entered by the user is "Chocolate B", the matching means 12b will determine that there is an inconsistency (mismatch) between these feature pieces of information. Then, the replacement means 12c will determine that "Chocolate A" is the correct feature piece and "Chocolate B" is the incorrect feature piece, based on the fact that "Chocolate A" is obtained from two modality data and "Chocolate B" is obtained from one modality data among the multiple feature pieces of information. Subsequently, the replacement means 12c can perform a replacement process to replace the incorrect feature piece "Chocolate B" with the correct feature piece "Chocolate A".

[0089] Furthermore, if multiple feature information such as product code, product name, and product category is obtained for the same product, the substitution means 12c may perform a majority vote for each feature information, or it may determine the correct feature information based on the result of a comprehensive evaluation of the consistency of the multiple feature information. In addition, if the result of the majority vote is a tie, the correct feature information may be determined based on a pre-set priority rule, the reliability of each modality data, the acquisition date and time, or the priority of the information source.

[0090] In this way, by performing a majority vote on feature information obtained from multiple modality data, even if some modality data contains erroneous feature information due to misrecognition, input errors, communication errors, or noise, the feature information common to all modality data can be determined as the correct feature information. As a result, the accuracy and reliability of the feature information can be improved, and subsequent processing based on erroneous feature information can be suppressed.

[0091] Furthermore, by cross-referencing feature information obtained from different sources, the impact of misrecognition and missing data can be reduced compared to relying solely on data from a single modality. This improves the consistency of product information, user information, object information, and other types of information, thereby enhancing the accuracy and reliability of information processing.

[0092] <System terminal / function / annotation method> Next, the annotation means 12d will be described.

[0093] The annotation means 12d is a means that can tag modality data using positive feature information stored in the storage device 26 by the replacement means 12c, and in this example, it is composed of a program stored in the storage device 26 of the system terminal 12, the storage device 26, etc.

[0094] For example, if the information storage terminal 14 stores the feature information (product price information) "3,820 yen" as positive feature information, the annotation means 12d uses the positive feature information ("Collagen-containing beauty serum, regular price 3,820 yen") to tag the audio data of the promotional video for the product (for example, "beauty serum", "collagen", "price: 3,820 yen"). This prevents incorrect tags from being assigned due to misrecognition of speech.

[0095] Furthermore, if the information storage terminal 14 stores the feature information "JAN:49013" (the product's JAN code) as positive feature information, the annotation means 12d uses the positive feature information ("JAN:49013") to attach tags (for example, "skincare," "lotion," "hyaluronic acid," "JAN:49013") to the image data of the product's packaging. This prevents incorrect tags from being attached, including misrecognitions in the image recognition process.

[0096] Furthermore, if the information storage terminal 14 stores the feature information (code assigned to the product) "490121" as positive feature information, the annotation means 12d uses the positive feature information ("490121") to assign tags (for example, "Bestselling product", "Purchased by housewives", "High demand on weekends", "Image & POS data consistency confirmed", "Code: 490121") to the shelf layout image data of the product. This makes it possible to effectively utilize the shelf layout image data in shelf layout optimization and sales promotion planning.

[0097] Furthermore, the system may include an evaluation means for evaluating positive feature information, and the annotation means 12d may only tag modality data using the positive feature information if the positive feature information meets predetermined criteria.

[0098] For example, if the replacement means 12c stores the JAN code of a certain product as the positive characteristic information of that product in the storage device 26, the evaluation means refers to knowledge acquired through machine learning or other means, or information obtained from external websites etc. via the network NW, and performs an evaluation by assigning an evaluation score on a scale of 1 to 10, for example, to the positive characteristic information (JAN code).

[0099] If the evaluation score of the positive feature information (JAN code) is above a predetermined standard (for example, 8 points) (i.e., if the confidence level of the positive feature information is high), the modality data is tagged using the positive feature information (JAN code). On the other hand, if the evaluation score of the positive feature information (JAN code) is below a predetermined standard (for example, 8 points) (i.e., if the confidence level of the positive feature information is low), the positive feature information (JAN code) is not used when tagging the modality data.

[0100] This configuration improves the quality of the feature information used when assigning tags, allowing for the assignment of appropriate tags to modality data.

[0101] Furthermore, the modality data includes data relating to the business information of the product seller and data relating to product purchase information, and the system is equipped with information analysis means capable of analyzing the business information, and the annotation means 12d may be capable of assigning different tags to a single purchase information depending on the results of the analysis of the business information when assigning tags.

[0102] For example, consider a case where acquisition means 12a performs barcode recognition processing on a shelf layout image of a product package and obtains "JAN:490120" as the image acquisition code, and also refers to sales performance data from the ID-POS system and obtains "JAN:490121" as POS character information, matching means 12b matches both codes and detects an inconsistency in the last digit, and replacement means 12c generates and stores ID-POS data "JAN:4901234567891" as correct feature information.

[0103] In this case, the information analysis means uses the correct JAN code, which has been identified as positive characteristic information, as a key to retrieve the business information of the product stored in the storage means 14, and performs, for example, the following three-axis analysis.

[0104] <Sales Period Analysis> Refer to annual sales data to identify periods with relatively high sales (e.g., December) and periods with low sales (e.g., July).

[0105] <Sales Location Analysis> Refer to the store's location information to determine whether it is an urban or suburban store.

[0106] <Competitive Analysis> Refer to external environmental data to determine whether there are any competing stores selling the same product.

[0107] What is important here is that the information analysis means can perform accurate business information analysis because the JAN code inconsistencies are resolved in advance by the matching means 12b and the substitution means 12c. If the business information analysis is performed with incorrect JAN codes, it will refer to the business information of a different product, and the analysis results and tag content will deviate from what was originally intended. In other words, the cross-modality consistency verification process functions as a prerequisite for the business information analysis process, and the two processes are technically inseparable.

[0108] Furthermore, the annotation means 12d combines the analysis results of the replacement character information and management information to assign multiple types of tags to the purchase information of the product, for example, as follows:

[0109] <Tags based on sales period analysis> "Sells well in winter" "Doesn't sell well in summer" <Tags based on sales location analysis> "Sells well even at high prices (urban areas)", "Does not sell well at high prices (suburban areas)" <Tags based on competitive analysis> "No competition, low price sensitivity", "Competition present, high price sensitivity" <Quality tag based on modality matching> "Image x POS data consistency confirmed" The annotation means 12d generates annotation information by attaching the above-mentioned multiple types of tags to a single piece of purchase information. The information processing system then selects the tag most suitable for a predetermined process (for example, forecasting year-end demand for products, forecasting sales for urban stores, or planning competitive countermeasures) from among the multiple types of tags included in the annotation information, and executes the predetermined process using the purchase information to which the tag has been attached.

[0110] As demonstrated in this example, an integrated processing configuration that starts with accurate purchasing information obtained through cross-modality consistency verification and then performs management information analysis enables highly accurate and multi-dimensional annotation that was not possible with conventional systems. In particular, in the retail industry, it is expected to be applied to sales promotion planning, demand forecasting, and shelf allocation management by integrating ID-POS data and product image / video data.

[0111] <Information Processing Systems / Summary> As described above, the information processing system according to this embodiment 1 (for example, the information processing system 10 shown in Figures 1 and 2) is an information processing system configured to have storage means (for example, storage means 14 shown in Figure 1, information storage terminal 14 shown in Figure 2) and an AI agent (for example, an AI agent 12 shown in Figure 1, system terminal 12 shown in Figure 2), wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is capable of acquiring feature information from the modality data (for example, acquisition means 12a shown in Figures 1 and 2), and acquires from at least two modality data. The information processing system is characterized by comprising: a matching means (for example, the matching means 12b shown in Figures 1 and 2) capable of matching multiple feature information and detecting inconsistencies between the multiple feature information; a replacement means (for example, the replacement means 12c shown in Figures 1 and 2) capable of performing a replacement process in which, when an inconsistency is detected, one of the multiple feature information is designated as correct feature information (hereinafter referred to as "correct feature information"), and a different feature information (hereinafter referred to as "incorrect feature information") is replaced with the correct feature information; and an annotation means (for example, the annotation means 12d shown in Figures 1 and 2) capable of tagging the modality data using the correct feature information.

[0112] Furthermore, the information processing method according to this embodiment 1 (for example, the method executed by the information processing system 10 shown in Figures 1 and 2) is an information processing method executed by an information processing system configured to have storage means (for example, storage means 14 shown in Figure 1, information storage terminal 14 shown in Figure 2) and an AI agent (for example, an AI agent 12 shown in Figure 1, a system terminal 12 shown in Figure 2), wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is capable of acquiring feature information from the modality data (for example, processing executed by acquisition means 12a shown in Figures 1 and 2), and acquiring multiple feature information from at least two modality data. The information processing method is characterized by being able to perform at least the following steps: a matching step (for example, a process performed by the matching means 12b shown in Figures 1 and 2) that can compare the multiple feature information and detect inconsistencies between them; a replacement step (for example, a process performed by the replacement means 12c shown in Figures 1 and 2) that, when an inconsistency is detected, sets one of the multiple feature information as correct feature information (hereinafter referred to as "correct feature information") and performs a replacement process that replaces feature information different from the correct feature information (hereinafter referred to as "incorrect feature information") with the correct feature information; and an annotation step (for example, a process performed by the annotation means 12d shown in Figures 1 and 2) that can tag the modality data using the correct feature information.

[0113] Furthermore, the information processing program according to this embodiment 1 (for example, a program executed by the information processing system 10 shown in Figures 1 and 2) is a program for an information processing system configured to have storage means (for example, storage means 14 shown in Figure 1, information storage terminal 14 shown in Figure 2) and an AI agent (for example, an AI agent 12 shown in Figure 1, system terminal 12 shown in Figure 2), wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is equipped with acquisition means (for example, acquisition means 12a shown in Figures 1 and 2) capable of acquiring characteristic information from the modality data, and at least two modality data This information processing program is characterized by comprising: a matching means (for example, the matching means 12b shown in Figures 1 and 2) capable of matching multiple feature information obtained from data and detecting inconsistencies between the multiple feature information; a replacement means (for example, the replacement means 12c shown in Figures 1 and 2) capable of performing a replacement process in which, when an inconsistency is detected, one of the multiple feature information is designated as the correct feature information (hereinafter referred to as "correct feature information"), and a feature information different from the correct feature information (hereinafter referred to as "incorrect feature information") is replaced with the correct feature information; and an annotation means (for example, the annotation means 12d shown in Figures 1 and 2) capable of tagging the modality data using the correct feature information.

[0114] According to the information processing system, information processing method, and information processing program of this embodiment 1, incorrect feature information can be corrected to correct feature information, tags can be assigned to modality data based on the correct feature information, and highly reliable and practical modality data can be generated.

[0115] Furthermore, the multiple feature information that the matching means matches may be feature information obtained from modality data of different types.

[0116] With this configuration, by comparing feature information obtained from modality data of different types, inconsistencies, misrecognitions, input errors, or missing information can be detected with higher accuracy compared to using only a single modality data set. Furthermore, since information contained in multiple modality data can be mutually complemented, the reliability and consistency of feature information can be improved. In addition, even if some modality data contains noise or misrecognitions, correction or verification can be performed using feature information obtained from other modality data, thereby improving the accuracy and robustness of information processing.

[0117] Furthermore, the system may include a means for assigning a mismatch score according to the degree of mismatch, and the replacement means may replace the erroneous feature information with the correct feature information only when the mismatch score exceeds a predetermined threshold.

[0118] This configuration allows for highly accurate automatic correction to correct feature information while suppressing the occurrence of incorrect substitutions.

[0119] Furthermore, the matching means may be capable of matching multiple feature information obtained from at least three modality data and detecting inconsistencies in the multiple feature information, and the replacement means may be capable of performing a replacement process in which, when an inconsistency is detected, the correct feature information is determined by majority vote from the multiple feature information and the incorrect feature information is replaced with the correct feature information.

[0120] With this configuration, by performing a majority vote on feature information obtained from multiple modality data, even if some modality data contains erroneous feature information due to misrecognition, input errors, communication errors, or noise, feature information common to multiple modality data can be determined as correct feature information. As a result, the accuracy and reliability of feature information can be improved, and subsequent processing based on erroneous feature information can be suppressed. In addition, by cross-referencing feature information obtained from different sources, the impact of misrecognition and missing data can be reduced compared to relying on only a single modality data. Therefore, the consistency of product information, user information, object information, and other various types of information can be improved, and the accuracy and reliability of information processing can be improved.

[0121] Furthermore, the system may include evaluation means for evaluating the positive feature information, and the annotation means may tag the modality data using the positive feature information only if the positive feature information meets predetermined criteria.

[0122] This configuration improves the quality of the replacement character information used when assigning tags, allowing for the assignment of appropriate tags to the data.

[0123] <Variation> Next, we will describe the modified information processing system 30 using Figure 3. Figure 3 is a schematic diagram showing an overview of the modified information processing system 30. To avoid redundant explanations, we will mainly describe the configuration that differs from the information processing system 10 described using Figures 1 and 2.

[0124] The modified information processing system 30 has a configuration in which, based on the feature information acquired by the acquisition means 12a as described in Figures 1 and 2, the context determination means 12e determines the processing context when assigning tags, and the annotation means 12f can assign different tags according to the determination result. In other words, even with the same feature information, the tags assigned will dynamically change if the processing context is different.

[0125] Here, "processing context" refers to information that indicates the purpose, usage scenario, processing content, or evaluation criteria when an AI agent tags feature information. Processing context is a general term for information that defines the scene, situation, and purpose when tagging feature information, and includes, for example, the following information.

[0126] (1) Information regarding timing: season, month, event, campaign status, etc. at the time of tagging, (2) Information regarding location: location of the store where the target data (modality data) was acquired (urban area, suburbs, etc.), sales channel (physical store, e-commerce site, etc.), market area characteristics, etc, (3) Information regarding purpose: type of processing to be performed after tagging (demand forecasting, sales analysis, shelf allocation optimization, competition (4) Information on users: Tag users (store staff, buyers, marketers, etc.), organizations (stores, headquarters, suppliers, etc.).

[0127] Other processing contexts include those related to product classification, product search, product recommendation, product display, inventory management, demand forecasting, sales analysis, customer analysis, marketing analysis, advertising distribution, anomaly detection, fraud detection, monitoring, quality control, maintenance management, business support, decision support, document classification, document search, summary generation, translation processing, patent search, patent analysis, patentability assessment, medical support, educational support, and dialogue support.

[0128] The context determination means 12e may acquire instruction information (such as prompts), search conditions, processing requests, or setting information entered by the user in order to obtain the processing context. For example, if the user instructs "Classify products into categories," the context determination means 12e will acquire the processing context related to product classification, and if the user instructs "Analyze the sales trends of products," it will acquire the processing context related to sales analysis.

[0129] Furthermore, the context determination means 12e may estimate the processing context based on the content of the modality data to be processed. For example, if the input data includes product information, it estimates the processing context related to product analysis, and if it includes data related to customer behavior, it estimates the processing context related to customer analysis.

[0130] Furthermore, the context determination means 12e may obtain the processing context based on the application, service, or workflow to be executed. For example, if executed within a product management application, it may obtain the processing context related to product management, and if executed within a marketing analysis application, it may obtain the processing context related to marketing analysis.

[0131] Furthermore, the context determination means 12e may determine the processing context based on the current time, location information, usage environment, connected system, or past processing history. For example, if accessed from a terminal within a store, it may obtain a processing context related to store operations, and if accessed from an administrator terminal, it may obtain a processing context related to management tasks. In addition, the context determination means 12e may generate multiple candidate processing contexts and select the optimal processing context based on the inference results of a machine learning model.

[0132] For example, the context determination means 12e refers to the management information, external environment data, processing request information, etc., stored in the storage means 14, and stores the result of determining the current processing context in the storage device 26. A specific processing example is shown below. It is assumed that the acquisition means 12a has acquired the purchase information (product ID, sales price) of product A as characteristic information.

[0133] (Case 1: When the purpose is demand forecasting) The context determination means 12e determines that the current processing objective is "year-end demand forecasting for product A". In this case, the annotation means 12f refers to the sales data of the management information and, based on the analysis result that the period when product A has the highest sales (December), assigns the tag "Sells well in winter" to the characteristic information (purchase information of product A).

[0134] (Case 2: When the purpose is to forecast sales for stores in urban areas) The context determination means 12e determines that the store being processed is located in an urban area as the current processing context. In this case, the annotation means 12f assigns the tag "Sells even at a high selling price" to the same characteristic information (purchase information for product A) based on the results of the store information analysis.

[0135] (Case 3: When the purpose is to plan countermeasures against competitors) The context determination means 12e determines that there are no competing stores in the current processing context. In this case, the annotation means 12f assigns the tag "no competition, low price sensitivity" to the same characteristic information (purchase information for product A) based on the results of the analysis of the competitor information.

[0136] As shown in cases 1-3 above, the same characteristic information (purchase information of product A) acquired by acquisition means 12a is assigned different tags depending on the processing context, such as "sells well in winter," "sells well even at high prices," and "no competition, low price sensitivity."

[0137] Furthermore, the annotation means 12f can generate annotation information in which tags corresponding to each of the above-mentioned multiple processing contexts are attached to a single piece of feature information. The processing execution means 12g selects the tag most suitable for the processing to be executed (demand forecasting, sales analysis, competitor countermeasures, etc.) from among the multiple tags included in the annotation information, and executes a predetermined processing using the feature information to which the selected tag is attached. This makes it possible to reuse annotation information generated in a single annotation process for multiple processing purposes, thereby increasing the efficiency of processing.

[0138] According to this modified method, dynamic tagging can be achieved according to the processing context, starting from feature information whose accuracy is ensured by cross-modality matching processing. In particular, in the retail industry, it becomes possible to dynamically assign tags for different business purposes, such as demand forecasting, sales analysis, shelf allocation optimization, and competitor countermeasures, to the same product purchase data, demonstrating high practical value in the integrated use of ID-POS data and product image and video data.

[0139] Furthermore, the processing context determined by the context determination means 12e may be dynamically determined based on the content of the processing request from the user of the external terminal 16, the type of processing performed by the internal scheduler, or the results of the analysis of management information stored in the storage means 14. This allows the information processing system 30 to autonomously determine the processing context and assign tags even if the user does not explicitly state the processing purpose.

[0140] Furthermore, the processing performed by the modified information processing system 30 is not limited to cases 1 to 3 above, but for example, cases 4 to 6 as follows are also conceivable.

[0141] (Case 4: Product Information Management) The AI ​​agent acquires characteristic information from multimodal data including product images, product descriptions, and ID-POS data. If the AI ​​agent determines that the processing context is for the purpose of analyzing product sales, it assigns tags indicating product categories such as "beverages," "confectionery," and "daily necessities." On the other hand, if the processing context is for the purpose of optimizing product display, it assigns tags indicating display strategies such as "facing the checkout," "facing the top shelf," and "potential for impulse purchases" to the same product.

[0142] (Case 5: Customer Behavior Analysis) The AI ​​agent acquires characteristic information from in-store camera video data, customer purchase history, location data, etc. If it determines that the processing context is for marketing analysis, it assigns tags indicating customer attributes such as "high-frequency visitor," "new customer," and "valuable customer." On the other hand, if it determines that the processing context is for security monitoring, it assigns monitoring tags such as "long-term stay," "normal behavior," and "behavior requiring verification" to the same customer behavior.

[0143] (Case 6: Document Analysis) The AI ​​agent acquires feature information from document data, image data, and audio data. If it determines that the processing context is for the purpose of searching for patent documents, it assigns tags indicating technical fields such as "communication technology," "image processing," and "machine learning." On the other hand, if it determines that the processing context is for the purpose of patentability evaluation, it assigns evaluation tags such as "novelty," "inventive step," and "enablement requirement" to the same feature information.

[0144] According to the modified information processing system, the AI ​​agent can flexibly utilize the same feature information for multiple purposes by assigning different tags to the same feature information depending on the processing context. Therefore, compared to assigning fixed tags in advance, it is possible to perform appropriate classification, searching, analysis, or inference according to the purpose of use.

[0145] Furthermore, because feature information can be interpreted and tagged from different perspectives depending on the processing context, the meaning and attributes contained in the feature information can be expressed more appropriately. As a result, the accuracy of information retrieval, analysis, inference, and decision-making can be improved.

[0146] Furthermore, since the same feature information can be reused according to different processing contexts, data management can be made more efficient compared to generating or managing separate feature information for each processing context. In addition, by automating tagging, manual classification work by users can be reduced, thereby lowering the operational burden.

[0147] In addition, because the AI ​​agent can dynamically change or add tags in response to changes in the processing context, it can flexibly adapt even when new usage purposes or analytical perspectives are added, thereby improving the system's scalability and adaptability.

[0148] <Information Processing Systems (Variations) / Summary> As described above, the modified information processing system (for example, the information processing system 30 shown in Figure 3) is an information processing system configured to have a storage means (for example, the storage means 14 shown in Figure 3) and an AI agent (for example, the AI ​​agent 32 shown in Figure 3), wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is characterized by comprising: an acquisition means (for example, the acquisition means 12a shown in Figure 3) capable of acquiring feature information from the modality data; a context determination means (for example, the context determination means 12e shown in Figure 3) capable of determining the processing context when assigning tags to the feature information; and an annotation means (for example, the annotation means 12f shown in Figure 3) capable of assigning different tags to the same feature information according to the processing context determined by the context determination means.

[0149] Furthermore, the modified information processing method is an information processing method performed by an information processing system (for example, the information processing system 30 shown in Figure 3) configured to have a storage means (for example, the storage means 14 shown in Figure 3) and an AI agent (for example, the AI ​​agent 32 shown in Figure 3), wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is capable of performing at least an acquisition step (for example, a process performed by the acquisition means 12a shown in Figure 3) capable of obtaining feature information from the modality data, a context determination step (for example, a process performed by the context determination means 12e shown in Figure 3) capable of determining the processing context when assigning tags to the feature information, and an annotation step (for example, a process performed by the annotation means 12f shown in Figure 3) capable of assigning different tags to the same feature information according to the processing context determined by the context determination step.

[0150] Furthermore, the modified information processing program is a program for an information processing system (for example, the information processing system 30 shown in Figure 3) configured to have a storage means (for example, the storage means 14 shown in Figure 3) and an AI agent (for example, the AI ​​agent 32 shown in Figure 3), wherein the storage means is capable of storing multimodal data including multiple modality data relating to the same object, and the AI ​​agent is configured to function as an acquisition means (for example, the acquisition means 12a shown in Figure 3) capable of acquiring feature information from the modality data, a context determination means (for example, the context determination means 12e shown in Figure 3) capable of determining the processing context when assigning tags to the feature information, and an annotation means (for example, the annotation means 12f shown in Figure 3) capable of assigning different tags to the same feature information according to the processing context determined by the context determination means.

[0151] According to the modified information processing system, information processing method, and information processing program, dynamic tagging according to the processing context can be achieved. In particular, in the retail industry, it becomes possible to dynamically assign tags for different business purposes, such as demand forecasting, sales analysis, shelf allocation optimization, and competitor countermeasures, to the same product purchase data, demonstrating high practical value in the integrated use of ID-POS data and product image and video data.

[0152] Furthermore, the processing context may include at least one of the following: information regarding when the data corresponding to the feature information was acquired; information regarding the location where the data corresponding to the feature information was acquired; and information regarding the purpose of the processing to be performed on the feature information.

[0153] Furthermore, the annotation means can generate annotation information in which multiple tags corresponding to multiple processing contexts are assigned to the same feature information, and the AI ​​agent may also include processing execution means (for example, processing execution means 12g shown in Figure 3, or processing execution means in Embodiment 3 described later) that can select the tag most suitable for the purpose of the processing to be performed from among the multiple tags included in the annotation information and execute a predetermined processing.

[0154] This configuration enhances the usefulness of tags used in specific processes (such as product sales analysis and demand forecasting), and allows for effective utilization of product purchase information during the execution of these processes.

[0155] Furthermore, a part of the configuration of the information processing system 10 according to Embodiment 1 may be combined with a part of the configuration of the modified information processing system 30. For example, the modified information processing system 30 may include a matching means capable of comparing a plurality of feature information obtained from at least two modality data and detecting inconsistencies in the plurality of feature information, and a replacement means capable of performing a replacement process in which, when an inconsistency is detected, one of the plurality of feature information is set as the correct feature information (positive feature information) and a different feature information (incorrect feature information) is replaced with the correct feature information, and the annotation means 12f may be capable of tagging the modality data using the positive feature information.

[0156] <<Embodiment 2>> Next, an information processing system 10 according to Embodiment 2 of the present invention will be described with reference to the drawings.

[0157] <Overview of the Information Processing System> First, an overview of the information processing system 10 according to this second embodiment will be described using Figure 4. Figure 4 is a schematic diagram showing an overview of the information processing system 10 according to this second embodiment.

[0158] The information processing system 10 according to this embodiment is an information processing system comprising a storage means 14 and an AI agent 12, wherein the storage means 14 is capable of storing annotation target data consisting of one or more of text data, image data, video data, and audio data, and the AI ​​agent 12 comprises an acquisition means 12a capable of acquiring character information contained in certain annotation target data, a replacement means 12b capable of performing a replacement process to replace noise containing ambiguous or inaccurate information in the character information with clear or accurate character information (hereinafter referred to as "replacement character information"), and an annotation means 12c capable of tagging certain annotation target data using the replacement character information.

[0159] According to the information processing system 10 of this second embodiment, even if the data to be tagged contains noise, an appropriate tag can be assigned to the data.

[0160] "Annotation target data" refers to data that will be tagged through annotation, and annotation target data consists of one or more of the following: text data, image data, video data, and audio data. Examples of "text data" include documents, emails, prompts, chat logs, papers, and program code. Examples of "image data" include photographs, drawings, blueprints, and graphs. Examples of "video data" include videos captured by cameras, commercials, promotional videos, and movies. Examples of "audio data" include prompts, conversations, music, instrument sounds, and animal sounds. "Textual information" refers to information that includes characters, numbers, and symbols.

[0161] "Ambiguous information" refers to unclear or ambiguous information, and in this example, AI agent 12 acquires "ambiguous information" through machine learning, etc. It should be noted that "ambiguous information" is not limited to simply unclear meanings, but also includes information whose meaning cannot be uniquely interpreted due to a lack of background knowledge, omission of logical structure, differences in cultural and intellectual backgrounds, etc.

[0162] Examples of "ambiguous information" include pronouns (e.g., that, this, it, over there, that), abbreviations (e.g., "U" for you, "Aimitsu" for multiple quotes), industry jargon used only in specific industries (e.g., "assign" in the hotel and inn industry refers to allocating various rooms to guests, but in the IT industry it refers to assigning personnel or tasks, and "Kranke" in hospitals refers to patients), and polysemous words with multiple meanings (e.g., "sweet" in chocolate can mean both "sweet" and "indulgent" towards a daughter, and the English word "book" can mean both "book" and "reservation").

[0163] "Inaccurate information" refers to information that is not accurate, and in this example, AI agent 12 acquires "inaccurate information" through machine learning, etc. Examples of "inaccurate information" include typos, omissions, mistranslations, grammatical errors, hallucinations (logical errors, facts created arbitrarily by the AI, misidentification of facts, etc.), and numerical errors.

[0164] Furthermore, the substitution means 12b may perform the substitution process using character information contained in annotation target data other than a certain annotation target data.

[0165] This configuration simplifies the replacement process and reduces the processing time required for replacing character information.

[0166] Furthermore, the system may include an evaluation means for evaluating replacement character information, and the annotation means 12c may, when the replacement character information meets a predetermined standard, use the replacement character information that meets the predetermined standard to tag a data to be annotated.

[0167] This configuration improves the quality of the replacement character information used when assigning tags, allowing for the assignment of appropriate tags to the data.

[0168] <Example of system configuration> Next, an example of the configuration of the information processing system 10 according to this second embodiment will be described using Figure 5. Figure 5 is a system configuration diagram showing an example of the configuration of the information processing system 10 according to this second embodiment.

[0169] The information processing system 10 can be configured, for example, to include a system terminal 12 that controls the entire system, and an information storage terminal 14 and an external terminal 16 that are connected to the system terminal 12 via a network NW so that they can communicate with each other.

[0170] The system terminal 12 is a terminal that controls the entire information processing system 10, and is composed of conventionally known servers, personal computers, etc. In this example, the system terminal 12 is composed of one server, but it may be composed of multiple servers, personal computers, etc. The hardware configuration of the system terminal 12 and the programs that the system terminal 12 executes will be described later.

[0171] The information storage terminal 14 is a terminal (storage means) for users (individuals or specific groups of multiple people (teams, departments, companies, etc.)) who use the information processing system 10 to store annotation target data, and is composed of conventionally known servers, personal computers, etc.

[0172] In this example, the information storage terminal 14 is configured with a single server, but it may be configured with multiple servers or personal computers. Furthermore, the terminal (storage means) for storing the annotation target data may be an internal storage means (for example, the storage device 26 shown in Figure 5) connected to the system terminal 12 via a local network (for example, LAN or P2P), or it may be other storage means.

[0173] External terminals 16 are terminals used by users of the information processing system 10 (individuals or specific groups of people (teams, departments, companies, etc.)), and consist of personal computers, tablets, smartphones, etc. The type of external terminal 16 is not particularly limited, but examples include smartphones, personal computers, and tablets used by individuals, and smartphones, personal computers, and tablets used by employees of companies, etc.

[0174] The network NW is a line that allows the system terminal 12, the information storage terminal 14, and the external terminal 16 to communicate with each other, and is typically composed of a WAN (Wide Area Network), also known as the Internet. The network NW may be wired or wireless, a LAN (Local Area Network), a dedicated line such as a VPN (Virtual Private Network), or a combination of these lines.

[0175] <System Terminal / Hardware Configuration Example> Next, we will describe an example of the hardware configuration of system terminal 12.

[0176] As shown in Figure 5, the system terminal 12 is configured to include, for example, a CPU 21 and a ROM 22, RAM 23, external storage drive 25, storage device 26, input device 27, display device 28, communication unit 29, etc., which are connected to the CPU 21 via a bus.

[0177] The CPU 21 is a control means that controls the entire system terminal 12, and performs processes such as executing application programs and operating systems (OS) stored in ROM 22 and storage devices 26, and storing data and files necessary for program execution in RAM 23 and storage devices 26.

[0178] ROM22 is a storage means for storing basic I / O programs and various data, and is composed of, for example, PROM, flash memory, etc. RAM23 is a storage means for temporarily storing data, and is composed of, for example, SDRAM, DRAM, etc. External storage drive25 is a control means that can read and write data to recording media 24 such as magnetic tape, DVD, etc., and is composed of, for example, magnetic tape storage, DVD drive, etc.

[0179] The storage device 26 is a storage means for storing application programs, the OS, control programs, related programs, various information, etc., and is composed of, for example, a hard disk drive (HDD), a solid-state drive (SDD), etc. The input device 27 is for inputting commands (instructions), etc., to the system terminal 12, and is composed of, for example, a keyboard, a pointing device (mouse, etc.), a touch panel, etc.

[0180] The display device 28 displays commands input by the input device 27, the response output of the system terminal 12 to those commands, and various other displays, and is composed of, for example, a liquid crystal display, plasma display, organic EL, etc. The communication unit 29 is a control means that controls communication with the information storage terminal 14 and external terminals 16, etc. via the network NW, and is composed of, for example, a communication card, etc.

[0181] <System Terminal / Function> Next, we will explain the functions of the system terminal 12.

[0182] The storage device 26 of the system terminal 12 stores a program (information processing program) that enables the system terminal 12 to function as an acquisition means 12a, a replacement means 12b, and an annotation means 12c.

[0183] <System terminal / Function / Acquisition method> Next, the acquisition means 12a will be described.

[0184] The acquisition means 12a is a means capable of acquiring character information (information including characters, numbers, and symbols) contained in certain annotation target data, and in this example, it is composed of a program stored in the storage device 26 of the system terminal 12, or the storage device 26, etc.

[0185] The acquisition means 12a, based on a specific trigger, refers to an annotation target data stored in the information storage terminal 14, acquires character information (information including characters, numbers, and symbols) from the annotation target data, and stores the acquired character information in the storage device 26 in association with the annotation target data.

[0186] The content of the "trigger" is not particularly limited and may be an externally provided trigger (e.g., a question (prompt) from a user of an external terminal 16, a signal input from an external sensor, or the receipt of an email from an external source) or an internally provided trigger (e.g., activation by an internal scheduler, occurrence of an anomaly, or update of specific information).

[0187] For example, if a new document file is stored in the information storage terminal 14 by a user of the external terminal 16 (if specific information is updated), the acquisition means 12a, triggered by the storage of a new document file in the information storage terminal 14, acquires character information (e.g., sentences, strings, numbers, codes) from the document file (annotation target data) using character recognition processing (OCR, etc.), and stores the character information in the storage device 26 in association with the document file.

[0188] Furthermore, for example, if a user of the external terminal 16 provides a voice question (prompt) (if a user of the external terminal 16 inputs a question (prompt)), the acquisition means 12a, triggered by the provision of a voice question (prompt), acquires text information (for example, the string of characters that make up the question, keywords of the question, etc.) from the voice data of the question (annotation target data) using speech recognition processing (ASR, etc.), and stores the text information in the storage device 26 in association with the voice data.

[0189] Furthermore, for example, when the update day (for example, once a week) arrives for updating the tags of the annotation target data (when triggered by the internal scheduler), the acquisition means 12a, triggered by the internal scheduler, acquires, for example, a photograph of a car as an arbitrary annotation target data from among the multiple annotation target data stored in the information storage terminal 14, acquires text information (for example, the name of the car, the manufacturer's name, the model) from the photograph of the car using character recognition processing (OCR, etc.), and stores the text information in the storage device 26 in association with the photograph of the car.

[0190] <System terminal / Function / Replacement method> Next, the substitution means 12b will be described.

[0191] The replacement means 12b is a means capable of performing a replacement process to replace (or modify, correct, or revise) the noise in the character information acquired by the acquisition means 12a with clear or accurate character information (replacement character information) when the noise contains ambiguous or inaccurate information. In this example, it is composed of a program stored in the storage device 26 of the system terminal 12, or the storage device 26, etc.

[0192] If the character information acquired by the acquisition means 12a contains noise that includes ambiguous or inaccurate information, the replacement means 12b uses knowledge acquired through machine learning or other means, or information acquired from external websites via the network NW, to replace the noise with clear or accurate character information (replacement character information), and stores the replacement character information in the storage device 26 in association with the data to be annotated.

[0193] For example, consider a case where character information (e.g., sentences, strings, numbers, codes) obtained from a document file by acquisition means 12a is stored in storage device 26, and this character information includes abbreviations (ambiguous information).

[0194] In this case, the replacement means 12b uses knowledge acquired through machine learning or other means, or information obtained from external websites via a network, to replace abbreviations (ambiguous information) contained in the text information with formal names (clear information), and stores the formal names (clear information) as replacement text information in the information storage terminal 14, associating them with the document file (annotation target data).

[0195] Furthermore, consider the case where, for example, character information (e.g., strings constituting the question, keywords of the question, etc.) acquired from the prompt's audio data by the acquisition means 12a is stored in the storage device 26, and this character information contains mistranslations (inaccurate information).

[0196] In this case, the replacement means 12b uses knowledge acquired through machine learning or other means, or information obtained from external websites via a network, to replace mistranslations (inaccurate information) contained in the text information with correct translations (accurate information), and stores the correct translations (accurate information) as replacement text information in the storage device 26, associating them with the prompt's audio data (annotation target data).

[0197] Furthermore, consider a case where, for example, textual information (e.g., car name, manufacturer name, model) acquired from a photograph of a car is stored in the information storage terminal 14 by the acquisition means 12a, and this textual information contains typographical errors (inaccurate information).

[0198] In this case, the replacement means 12b uses knowledge acquired through machine learning or other means, or information obtained from external websites via a network, to replace misspellings (inaccurate information) contained in the character information with correct characters (accurate information), and stores the correct characters (accurate information) as replacement character information in the storage device 26, associating them with the photograph of the automobile (annotation target data).

[0199] The replacement means 12b may also perform the replacement process using character information contained in annotation target data other than a certain annotation target data.

[0200] For example, when the replacement means 12b replaces abbreviations (ambiguous information) contained in the character information of a document file with formal names (clear information), it may obtain text information (e.g., formal names) from other annotation target data (e.g., other document files) stored in the information storage terminal 4 and use this text information (formal names) to perform the replacement of the character information.

[0201] This configuration simplifies the replacement process and reduces the processing time required for replacing character information.

[0202] <System terminal / function / annotation method> Next, the annotation means 12c will be described.

[0203] Annotation means 12c is a means that can tag a data to be annotated using replacement character information stored in the storage device 26 by replacement means 12b. In this example, it is composed of a program stored in the storage device 26 of the system terminal 12, and the storage device 26, etc.

[0204] The annotation means 12c uses the replacement character information stored in the storage device 26 by the replacement means 12b to tag a data to be annotated.

[0205] For example, if the official name (clear information) is stored in the information storage terminal 14 as replacement character information for a document file (data to be annotated), the annotation means 12c will use the official name (clear information) to tag the document file and store the tag in the information storage terminal 14 in association with the document file.

[0206] This avoids using abbreviations (ambiguous information) to tag document files, and allows for the assignment of appropriate tags to document files (annotation target data) even if the document files to be tagged (annotation target data) contain noise (ambiguous information).

[0207] Furthermore, for example, if the information storage terminal 14 stores the correct translation (accurate information) as replacement character information for the prompt's audio data (annotation target data), the annotation means 12c uses the correct translation (accurate information) to tag the document file and stores the tag in the information storage terminal 14 in association with the prompt's audio data.

[0208] This prevents the use of mistranslations (inaccurate information) to tag prompt audio data, and allows for the appropriate tags to be assigned to prompt audio data (annotation target data) even if the prompt audio data to be tagged (annotation target data) contains noise (inaccurate information).

[0209] Furthermore, for example, if the information storage terminal 14 stores the correct characters (accurate information) as replacement character information for the photograph of the automobile (annotation target data), the annotation means 12c will use the correct characters (accurate information) to tag the photograph of the automobile and store the tag in the information storage terminal 14 in association with the photograph of the automobile.

[0210] This prevents the use of typographical errors (inaccurate information) to tag car photos, and ensures that appropriate tags are assigned to car photos (annotation data) even if the car photos to be tagged (annotation data) contain noise (inaccurate information).

[0211] Furthermore, the annotation means 12c may include an evaluation means for evaluating replacement character information, and if the replacement character information meets a predetermined standard, it may use the replacement character information that meets the predetermined standard to tag a data to be annotated.

[0212] For example, if the replacement means 12b stores the correct translation (accurate information) in the storage device 26 as replacement character information for a mistranslation (inaccurate information) contained in the character information of the prompt's voice data, the evaluation means evaluates the correct translation (accurate information) by referring to knowledge acquired through machine learning or other means, or information obtained from external websites etc. via the network NW, and assigning an evaluation score on a scale of 1 to 10, for example.

[0213] Then, if the evaluation score of the replacement character information (in this example, the correct translation) is equal to or above a predetermined standard (for example, 8 points), the replacement character information (correct translation) is used to tag a certain annotation target data. On the other hand, if the evaluation score of the replacement character information (correct translation) is below a predetermined standard (for example, 8 points), the replacement character information (correct translation) is not used when tagging a certain annotation target data.

[0214] This configuration improves the quality of the replacement character information used when assigning tags, allowing for the assignment of appropriate tags to the data.

[0215] <Information Processing Systems / Summary> As described above, the information processing system according to this second embodiment (for example, the information processing system 10 shown in Figures 4 and 5) is an information processing system configured to have storage means (for example, storage means 14 shown in Figure 4, information storage terminal 14 shown in Figure 5) and an AI agent (for example, an AI agent 12 shown in Figure 4, system terminal 12 shown in Figure 5), wherein the storage means is capable of storing annotation target data consisting of one or more of text data, image data, video data, and audio data, and the AI ​​agent is included in certain annotation target data. The information processing system is characterized by comprising: an acquisition means (for example, an acquisition means 12a shown in Figures 4 and 5) capable of acquiring character information; a replacement means (for example, a replacement means 12b shown in Figures 4 and 5) capable of performing a replacement process to replace the noise containing ambiguous or inaccurate information in the character information with clear or accurate character information (hereinafter referred to as "replacement character information"); and an annotation means (for example, an annotation means 12c shown in Figures 4 and 5) capable of tagging a certain annotation target data using the replacement character information.

[0216] Furthermore, the information processing method according to this second embodiment (for example, the method executed by the information processing system 10 shown in Figures 4 and 5) is an information processing method executed by an information processing system configured to have storage means (for example, storage means 14 shown in Figure 4, information storage terminal 14 shown in Figure 5) and an AI agent (for example, an AI agent 12 shown in Figure 4, a system terminal 12 shown in Figure 5), wherein the storage means is capable of storing annotation target data consisting of one or more of text data, image data, video data, and audio data, and the AI ​​agent acquires character information contained in certain annotation target data. The information processing method is characterized by being able to perform at least the following steps: an acquisition step (for example, a process performed by the acquisition means 12a shown in Figures 4 and 5); a replacement step (for example, a process performed by the replacement means 12b shown in Figures 4 and 5) that performs a replacement process to replace the noise containing ambiguous or inaccurate information in the character information with clear or accurate character information (hereinafter referred to as "replacement character information"); and an annotation step (for example, a process performed by the annotation means 12c shown in Figures 4 and 5) that can tag a certain annotation target data using the replacement character information.

[0217] Furthermore, the information processing program according to this second embodiment (for example, a program executed by the information processing system 10 shown in Figures 4 and 5) is a program for an information processing system configured to have storage means (for example, storage means 14 shown in Figure 4, information storage terminal 14 shown in Figure 5) and an AI agent (for example, an AI agent 12 shown in Figure 4, system terminal 12 shown in Figure 5), wherein the storage means is capable of storing annotation target data consisting of one or more of text data, image data, video data, and audio data, and the AI ​​agent is configured to handle a certain annotation target data This information processing program is characterized by comprising: an acquisition means (for example, an acquisition means 12a shown in Figures 4 and 5) capable of acquiring character information contained in data; a replacement means (for example, a replacement means 12b shown in Figures 4 and 5) capable of performing a replacement process to replace noise containing ambiguous or inaccurate information in the character information with clear or accurate character information (hereinafter referred to as "replacement character information"); and an annotation means (for example, an annotation means 12c shown in Figures 4 and 5) capable of assigning tags to a certain annotation target data using the replacement character information.

[0218] According to the information processing system, information processing method, and information processing program of this second embodiment, even if the data to be tagged contains noise, appropriate tags can be assigned to the data.

[0219] Furthermore, the substitution means (or the substitution step) may perform the substitution process using character information contained in annotation target data other than the aforementioned annotation target data.

[0220] This configuration simplifies the replacement process and reduces the processing time required for replacing character information.

[0221] Furthermore, the system may include an evaluation means (or evaluation step) for evaluating the replacement character information, and the annotation means (or annotation step) may, if the replacement character information satisfies predetermined criteria, use the replacement character information that satisfies the predetermined criteria to tag a certain annotation target data.

[0222] This configuration improves the quality of the replacement character information used when assigning tags, allowing for the assignment of appropriate tags to the data.

[0223] <<Embodiment 3>> Next, an information processing system 10 according to Embodiment 3 of the present invention will be described with reference to the drawings.

[0224] <Overview of the Information Processing System> First, an overview of the information processing system 10 according to this third embodiment will be described using Figure 6. Figure 6 is a schematic diagram showing an overview of the information processing system 10 according to this third embodiment.

[0225] The information processing system 10 according to this embodiment is an information processing system comprising a storage means 14 and an AI agent 12, wherein the storage means 14 is capable of storing business information of a product seller and product purchase information, and the AI ​​agent 12 comprises an information analysis means 12a capable of analyzing business information and an annotation means 12b capable of assigning tags to purchase information based on the analysis results of the business information, wherein the annotation means 12b is capable of assigning different tags to a single piece of purchase information depending on the analysis results of the business information when assigning tags.

[0226] According to the information processing system 10 of this third embodiment, the usefulness of tags attached to product purchase information can be enhanced, and product purchase information can be effectively utilized in commercial activities such as product sales analysis and demand forecasting.

[0227] Here, "management information" refers to the data and knowledge necessary for a company to operate efficiently and effectively. Examples of "management information" include management figures, customer information, financial information, market information, and competitor information.

[0228] "Management numerical information" refers to numerical information that visualizes a company's management status through management indicators and financial indicators, and is used to evaluate company performance. Examples of "management numerical information" include: (1) sales and profit-related data (e.g., sales data, profit data), (2) sales and product management-related data (e.g., product master data, specifications, shelf layout data, sales performance and promotional effectiveness), (3) inventory and purchasing-related data (e.g., inventory status, ordering and purchasing data), (4) customer and marketing-related data (e.g., customer ID data (ID-POS), marketing campaign results), (5) store operation and personnel management-related data (e.g., store location, store performance, staff shifts), (6) expense and financial-related data (e.g., store and headquarters expenses, financial indicators), (7) external environment data (e.g., trade area / population data, competitor store location and pricing information, weather, temperature, and disaster information), (8) local event / school event information, and (9) instruction and communication-related data (e.g., work instructions and policies from superiors, reports and suggestions from the field, history of collaboration with headquarters).

[0229] In the case of companies in the retail industry, etc., (2) Sales and product management related data (e.g., product master data, specifications, shelf layout data, sales performance and promotional effects) within "management numerical information" may be referred to as "sales promotion information," and (4) Customer and marketing related data (e.g., customer ID data (ID-POS), results of marketing measures) within "management numerical information" may be referred to as "purchase history information."

[0230] "Purchasing information" refers to information such as product name, price, delivery date, and conditions that a company needs when purchasing goods from a supplier. Examples of "purchasing information" include product information (product ID, product name, category (JAN code), brand, unit price, etc.), order quantity, and delivery date.

[0231] Furthermore, management information may include information regarding the timing of product sales (e.g., sales figures, profit figures, event and campaign information), information regarding product sales locations (e.g., store information), and information regarding product sales scenarios (e.g., competitor information, market information).

[0232] This configuration allows for an increase in the variety of tags attached to product purchase information, thereby enhancing the usefulness of the tags.

[0233] Furthermore, the annotation means 12b may refer to product sales information and assign different tags to a single purchase information depending on the sales status of the product at the time of tagging.

[0234] With this configuration, different tags can be assigned to a single purchase information item depending on the sales status of the product (for example, the relationship between product sales and sales timing), thereby increasing the usefulness of the tags.

[0235] Furthermore, the annotation means 12b may refer to information about the store where the product is sold and assign different tags to a single purchase information depending on the information of the store where the tags are assigned.

[0236] With this configuration, different tags can be assigned to a single purchase record depending on the information of the store where the product is sold (for example, differences in the location where the product is sold), thereby increasing the usefulness of the tags.

[0237] Furthermore, the annotation means 12b may refer to competitive information on products and assign different tags to a single purchase information depending on the competitive information when assigning tags.

[0238] With this configuration, different tags can be assigned to a single purchase data item depending on the competitive information of the product (differences in the sales scenario of the product), thereby increasing the usefulness of the tags.

[0239] Furthermore, the system may include a processing execution means capable of performing predetermined processing using tagged purchase information, the annotation means 12b capable of generating annotation information with multiple types of tags attached to a single purchase information, and the processing execution means selects one tag from among the multiple types of tags included in the annotation information that is most suitable for the predetermined processing, and performs the predetermined processing using the purchase information to which that tag is attached.

[0240] This configuration enhances the usefulness of tags used in specific processes (such as product sales analysis and demand forecasting), and allows for effective utilization of product purchase information during the execution of these processes.

[0241] <Example of system configuration> Next, an example of the configuration of the information processing system 10 according to this embodiment 3 will be described using Figure 7. Figure 7 is a system configuration diagram showing an example of the configuration of the information processing system 10 according to this embodiment 3.

[0242] The information processing system 10 can be configured, for example, to include a system terminal 12 that controls the entire system, and an information storage terminal 14 and an external terminal 16 that are connected to the system terminal 12 via a network NW so that they can communicate with each other.

[0243] The system terminal 12 is a terminal that controls the entire information processing system 10, and is composed of conventionally known servers, personal computers, etc. In this example, the system terminal 12 is composed of one server, but it may be composed of multiple servers, personal computers, etc. The hardware configuration of the system terminal 12 and the programs that the system terminal 12 executes will be described later.

[0244] The information storage terminal 14 is a terminal (storage means) used by a retail store (e.g., a drugstore, retail store, convenience store, etc.) to store management information and product purchase information of the store, and is composed of a POS terminal, a conventionally known server, a personal computer, etc.

[0245] As mentioned above, "management information" refers to the data and knowledge necessary for a company to operate efficiently and effectively. Examples of "management information" include management figures, customer information, financial information, market information, and competitor information.

[0246] In this example, the information storage terminal 14 is configured with a single server, but it may also be configured with multiple servers or personal computers. Furthermore, the terminal (storage means) for storing store management information and product purchase information may be an internal storage means (for example, the storage device 26 shown in Figure 7) connected to the system terminal 12 via a local network (for example, LAN or P2P), or it may be other storage means.

[0247] External terminals 16 are terminals used by users of the information processing system 10, and consist of personal computers, tablets, smartphones, etc. The type of external terminal 16 is not particularly limited, but examples include smartphones, personal computers, and tablets used by individuals, as well as smartphones, personal computers, and tablets used by employees of sales outlets, etc.

[0248] The network NW is a line that allows the system terminal 12, the information storage terminal 14, and the external terminal 16 to communicate with each other, and is typically composed of a WAN (Wide Area Network), also known as the Internet. The network NW may be wired or wireless, a LAN (Local Area Network), a dedicated line such as a VPN (Virtual Private Network), or a combination of these lines.

[0249] <System Terminal / Hardware Configuration Example> Next, we will describe an example of the hardware configuration of system terminal 12.

[0250] As shown in Figure 7, the system terminal 12 is configured to include, for example, a CPU 21, and a ROM 22, RAM 23, external storage drive 25, storage device 26, input device 27, display device 28, communication unit 29, etc., all connected to the CPU 21 via a bus.

[0251] The CPU 21 is a control means that controls the entire system terminal 12, and performs processes such as executing application programs and operating systems (OS) stored in ROM 22 and storage devices 26, and storing data and files necessary for program execution in RAM 23 and storage devices 26.

[0252] ROM22 is a storage means for storing basic I / O programs and various data, and is composed of, for example, PROM, flash memory, etc. RAM23 is a storage means for temporarily storing data, and is composed of, for example, SDRAM, DRAM, etc. External storage drive25 is a control means that can read and write data to recording media 24 such as magnetic tape, DVD, etc., and is composed of, for example, magnetic tape storage, DVD drive, etc.

[0253] The storage device 26 is a storage means for storing application programs, the OS, control programs, related programs, various information, etc., and is composed of, for example, a hard disk drive (HDD), a solid-state drive (SDD), etc. The input device 27 is for inputting commands (instructions), etc., to the system terminal 12, and is composed of, for example, a keyboard, a pointing device (mouse, etc.), a touch panel, etc.

[0254] The display device 28 displays commands input by the input device 27, the response output of the system terminal 12 to those commands, and various other displays, and is composed of, for example, a liquid crystal display, plasma display, organic EL, etc. The communication unit 29 is a control means that controls communication with the information storage terminal 14 and external terminals 16, etc. via the network NW, and is composed of, for example, a communication card, etc.

[0255] <System Terminal / Function> Next, we will explain the functions of the system terminal 12.

[0256] The storage device 26 of the system terminal 12 stores a program (information processing program) that enables the system terminal 12 to function as an information analysis means 12a and an annotation means 12b.

[0257] <System terminal / function / information analysis means> Next, the information analysis means 12a will be described.

[0258] The information analysis means 12a is a means capable of analyzing the business information of the product seller, and in this example, it consists of a program stored in the storage device 26 of the system terminal 12, and the storage device 26, etc.

[0259] The information analysis means 12a acquires management information stored in the information storage terminal 14, performs analysis of the acquired management information, and stores the analysis results in the storage device 26.

[0260] For example, consider a case where the information analysis means 12a refers to sales and profit-related data stored in the information storage terminal 14 as information regarding the timing of product sales, and obtains the annual sales data (product sales information) for product A from this sales and profit-related data.

[0261] In this case, the information analysis means 12a analyzes the annual sales data (sales information of the product) of product A and extracts, for example, the month in which product A has the highest sales (for example, December) and the month in which product A has the lowest sales (for example, July). Then, the month in which product A has the highest sales (December) is stored in the storage device 26 as information about "a period when product A has relatively good sales," and the month in which product A has the lowest sales (July) is stored in the storage device 26 as information about "a period when product A has relatively bad sales."

[0262] Furthermore, the analysis of "information regarding the timing of product sales" is not limited to product sales information, but may also include, for example, product profit information (e.g., whether or not there is a profit, and the magnitude of the profit), event and campaign information (e.g., whether or not an event or campaign was held, the number of times it was held, and its duration), etc.

[0263] Furthermore, consider a case where, for example, the information analysis means 12a refers to store operation and personnel management related data stored in the information storage terminal 14 as information about the location where the product is sold, and obtains information about store B that sells product A (information about the store where the product is sold) from this store operation and personnel management related data.

[0264] In this case, the information analysis means 12a analyzes the information of store B that sells product A (information of the store selling the product) and determines, for example, whether the location of store B is in an urban area or outside an urban area. If it is determined that the location of store B is in an urban area, it stores information indicating that "store B is located in an urban area" in the storage device 26 as information of store B that sells product A (information of the store selling the product). If it is determined that the location of store B is outside an urban area, it stores information indicating that "store B is located outside an urban area" in the storage device 26 as information of store B that sells product A (information of the store selling the product).

[0265] Furthermore, the analysis of "information regarding the sales location of products" is not limited to information on the location of stores where products are sold. For example, it may also include information on the sales method of the product (sales at physical stores or sales on e-commerce sites), information on e-commerce sites, whether or not there are overseas stores, and the locations of overseas stores.

[0266] Furthermore, consider a case where, for example, the information analysis means 12a refers to external environmental data stored in the information storage terminal 14 as information related to the sales scene (sales situation) of the product, and obtains information about competing stores that sell product A (product competition information) from this external environmental data.

[0267] In this case, the information analysis means 12a analyzes information about competing stores that sell product A (product competition information) and determines, for example, whether or not there are competing stores that sell product A. If it determines that there are competing stores that sell product A, it stores information indicating that "there are competing stores that sell product A" in the storage device 26 as information about competing stores that sell product A (product competition information). If it determines that there are no competing stores that sell product A, it stores information indicating that "there are no competing stores that sell product A" in the storage device 26 as information about competing stores that sell product A (product competition information).

[0268] Furthermore, the analysis of "information regarding the sales scenario of a product" is not limited to competitive information on the product, but may also include, for example, market information on the product (e.g., market share, demand trends, consumer needs, growth rate, trends, etc.).

[0269] <System terminal / function / annotation method> Next, the annotation means 12b will be described.

[0270] Annotation means 12b is capable of tagging purchase information based on the analysis results of management information generated by information analysis means 12a, and is capable of assigning different tags to a single purchase information depending on the analysis results of management information when tagging. In this example, it is composed of a program stored in the storage device 26 of the system terminal 12, and the storage device 26, etc.

[0271] The annotation means 12b refers to the results of the analysis of management information at the time of tagging purchase information related to a product (for example, product ID and sales price), and assigns different tags to a single piece of purchase information according to the results of the analysis of management information.

[0272] For example, consider a case where, as a result of the analysis of management information by the information analysis means 12a, the month in which product A has the highest sales (December) is stored in the memory device 26 as information indicating "a period when product A has relatively good sales," and the month in which product A has the lowest sales (July) is stored in the memory device 26 as information indicating "a period when product A has relatively poor sales."

[0273] In this case, the annotation unit 12b refers to the analysis result of the management information at the timing of assigning a tag to the purchase information related to product A (for example, the ID and sales price of product A), and when the month with the highest sales of product A (December) is stored in the storage device 26 as the analysis result of the management information, since product A tends to sell well even in winter, a tag with the content "sells well in winter" is assigned to the purchase information related to product A (for example, the ID and sales price of product A) as shown in Fig. 8(a).

[0274] On the other hand, the annotation unit 12b refers to the analysis result of the management information at the timing of assigning a tag to the purchase information related to product A (for example, the ID and sales price of product A), and when the month with the lowest sales of product A (July) is stored in the storage device 26 as the analysis result of the management information, since product A tends not to sell well in summer, a tag with the content "does not sell well in summer" is assigned to the purchase information related to product A (for example, the ID and sales price of product A) as shown in Fig. 8(a).

[0275] That is, the annotation unit 12b refers to the sales information of the product, and assigns different tags (in this example, sells well in winter or does not sell well in summer) to one piece of purchase information (in this example, the ID and sales price of product A) according to the sales status of the product at the time of tag assignment (in this example, the relationship between sales period and sales volume).

[0276] According to this example, different tags can be assigned to one piece of purchase information according to the sales status of the product (for example, the difference in the sales period of the product), and the usefulness of the tags can be improved.

[0277] Further, for example, consider a case where, as a result of analysis of management information by the information analysis unit 12a, information indicating that "the location of store B is in an urban area" or information indicating that "the location of store B is in a place other than an urban area" is stored in the storage device 26 as the information of store B that sells product A (the information of the product sales store).

[0278] In this case, the annotation means 12b refers to the results of the business information analysis when tagging the purchase information related to product A (for example, the ID and selling price of product A). If the analysis results of the business information indicate that "store B is located in an urban area," then, since products tend to sell even if the selling price is high in urban areas, the annotation means 12b adds a tag with the content "sells even if the selling price is high" to the purchase information related to product A (for example, the ID and selling price of product A), as shown in Figure 8(b).

[0279] On the other hand, the annotation means 12b refers to the results of the business information analysis when tagging the purchase information related to product A (for example, the ID and selling price of product A). If the analysis results of the business information indicate that "store B is located outside of urban areas," then, since products tend not to sell well outside of urban areas if the selling price is high, the annotation means 12b adds a tag with the content "does not sell well if the selling price is high" to the purchase information related to product A (for example, the ID and selling price of product A), as shown in Figure 8(b).

[0280] In other words, annotation means 12b refers to information about the store where the product is sold and assigns different tags (in this example, the ID of product A and the selling price) to a single piece of purchase information (in this example, the ID of product A and the selling price) depending on the differences in the store information (in this example, the location of the store) when assigning tags.

[0281] In this example, different tags can be assigned to a single purchase record depending on the information of the store where the product is sold (for example, differences in the location where the product is sold), thereby increasing the usefulness of the tags.

[0282] Furthermore, consider a case where, as a result of the analysis of management information by the information analysis means 12a, information indicating that "there are competing stores that sell product A" or "there are no competing stores that sell product A" is stored as information about competing stores that sell product A (product competition information).

[0283] In this case, the annotation means 12b refers to the results of the business information analysis when tagging the purchase information related to product A (for example, the ID and selling price of product A). If the analysis results of the business information indicate that "there are competing stores that sell product A," then it is predicted that product A will not sell well if there are many competing stores and the selling price is high. Therefore, as shown in Figure 8(c), the annotation means 12b adds a tag with the content "will not sell if the selling price is high" to the purchase information related to product A (for example, the ID and selling price of product A).

[0284] On the other hand, the annotation means 12b refers to the results of the business information analysis when tagging purchase information related to product A (for example, the ID and selling price of product A). If the analysis results of the business information indicate that "there are no competing stores selling product A" and this information is stored in the storage device 26, then it is predicted that product A will sell even if the selling price is high because there are no competing stores. Therefore, as shown in Figure 8(c), the annotation means 12b adds a tag with the content "will sell even if the selling price is high" to the purchase information related to product A (for example, the ID and selling price of product A).

[0285] In other words, the annotation means 12b refers to the competitive information of the product and assigns different tags (in this example, the ID and selling price of product A) to a single piece of purchase information (in this example, the ID and selling price of product A) according to the competitive information of the product when assigning tags.

[0286] In this example, different tags can be assigned to a single purchase data item depending on the competitive information of the product (differences in the sales scenario of the product), thereby increasing the usefulness of the tags.

[0287] <System terminal / Function / Processing execution means> Next, the processing execution means will be described.

[0288] The processing execution means is a means capable of executing predetermined processing using tagged purchase information, and in this example, it consists of a program stored in the storage device 26 of the system terminal 12, and the storage device 26, etc.

[0289] The processing execution means selects one tag from among the multiple types of tags contained in annotation information (for example, information consisting of a combination of one "purchase information" and multiple types of "tags" as shown in Figures 8(a) to 8(c)) to which one purchase information has been assigned multiple types of tags, which is most suitable for a predetermined process, and executes the predetermined process using the purchase information to which that tag has been assigned.

[0290] The content of "prescribed processing" is not particularly limited and could include, for example, sales analysis of products or demand forecasting.

[0291] For example, when the processing execution means performs a year-end demand forecast for a product as a predetermined process, it selects one tag (in this example, "Sells well in winter") from among the multiple types of tags (in this example, "Sells well in winter" or "Sells poorly in summer") included in the annotation information shown in Figure 8(a) that is most suitable for the predetermined process (year-end demand forecast for the product), and uses the purchase information to which that tag is attached (in this example, the ID and selling price of product A) to perform the predetermined process (year-end demand forecast for the product).

[0292] Furthermore, for example, if the processing execution means performs a predetermined process, such as forecasting sales for stores located in urban areas, it selects one tag (in this example, "sells even if the selling price is high") from among the multiple types of tags (in this example, "sells even if the selling price is high") included in the annotation information shown in Figure 8(b) that is best suited to the predetermined process (forecasting sales for stores located in urban areas), and uses the purchase information to which that tag is attached (in this example, the ID and selling price of product A) to execute the predetermined process (forecasting sales for stores located in urban areas).

[0293] Further, for example, when the processing execution means executes sales prediction for products with less competition as the predetermined processing, from among multiple types of tags (in this example, "does not sell if the selling price is high" or "sells even if the selling price is high") included in the annotation information shown in Figure 8(c), the one tag most suitable for the predetermined processing (sales prediction for products with less competition) (in this example, "sells even if the selling price is high") is selected, and the predetermined processing (sales prediction for products with less competition) is executed using the purchase information to which the tag is assigned (in this example, the ID and selling price of Product A).

[0294] According to this example, in predetermined processing (for example, product sales analysis, demand prediction, etc.) Use The usefulness of the tags can be enhanced, and the product purchase information can be effectively utilized in the process of executing the predetermined processing.

[0295] <Information Processing System / Summary> As described above, the information processing system according to the third embodiment (for example, the information processing system 10 shown in Figures 6 and 7) is an information processing system configured to comprise a storage means (for example, the storage means 14 shown in Figure 6, the information storage terminal 14 shown in Figure 7) and an AI agent (for example, the AI agent 12 shown in Figures 6 and 7), wherein the storage means is capable of storing management information of product sellers and product purchase information, the AI agent is provided with an information analysis means (for example, the information analysis means 12a shown in Figures 6 and 7) capable of analyzing the management information, and an annotation means (for example, the annotation means 12b shown in Figures 6 and 7) capable of assigning tags to the purchase information based on the analysis result of the management information, and the annotation means is capable of assigning different tags to one piece of the purchase information according to the analysis result of the management information at the time of assigning the tags.

[0296] Furthermore, the information processing method according to this third embodiment (for example, the method executed by the information processing system 10 shown in Figures 6 and 7) is an information processing method executed by an information processing system configured to have storage means (for example, storage means 14 shown in Figure 6, information storage terminal 14 shown in Figure 7) and an AI agent (for example, an AI agent 12 shown in Figures 6 and 7), wherein the storage means is capable of storing business information of the product seller and product purchase information, and the AI ​​agent is capable of at least performing an information analysis step capable of analyzing the business information (for example, a process executed by the information analysis means 12a shown in Figures 6 and 7) and an annotation step capable of assigning tags to the purchase information based on the analysis results of the business information (for example, a process executed by the annotation means 12b shown in Figures 6 and 7), wherein the annotation step is capable of assigning different tags to a single piece of purchase information depending on the analysis results of the business information when assigning the tags.

[0297] Furthermore, the information processing program according to this third embodiment (for example, a program executed by the information processing system 10 shown in Figures 6 and 7) is a program for an information processing system configured to have storage means (for example, storage means 14 shown in Figure 6, information storage terminal 14 shown in Figure 7) and an AI agent (for example, an AI agent 12 shown in Figures 6 and 7), wherein the storage means is capable of storing business information of the product seller and product purchase information, and the AI ​​agent functions as information analysis means (for example, information analysis means 12a shown in Figures 6 and 7) capable of analyzing the business information and annotation means (for example, annotation means 12b shown in Figures 6 and 7) capable of assigning tags to the purchase information based on the analysis results of the business information, and the annotation means is capable of assigning different tags to a single piece of purchase information depending on the analysis results of the business information when assigning the tags.

[0298] According to the information processing system, information processing method, and information processing program of this third embodiment, the usefulness of tags attached to product purchase information can be enhanced, and product purchase information can be effectively utilized in commercial activities such as product sales analysis and demand forecasting.

[0299] Furthermore, the aforementioned management information may include information regarding the timing of product sales, information regarding product sales locations, and information regarding product sales scenarios.

[0300] This configuration allows for an increase in the variety of tags attached to product purchase information, thereby enhancing the usefulness of the tags.

[0301] Furthermore, the information regarding the sales period of the product may include sales information, the information regarding the sales location of the product may include information about the stores where the product is sold, and the information regarding the sales scenario of the product may include information about competitors for the product.

[0302] This configuration allows for an increase in the variety of tags attached to product purchase information, thereby enhancing the usefulness of the tags.

[0303] Furthermore, the annotation means may refer to the sales information of the product and assign different tags to the purchase information depending on the sales status of the product at the time of tag assignment.

[0304] With this configuration, different tags can be assigned to a single purchase information item depending on the sales status of the product (for example, the relationship between product sales and sales timing), thereby increasing the usefulness of the tags.

[0305] Furthermore, the annotation means may refer to information about the sales store of the product and assign different tags to the purchase information according to the sales store information when assigning the tags.

[0306] With this configuration, different tags can be assigned to a single purchase record depending on the information of the store where the product is sold (for example, differences in the location where the product is sold), thereby increasing the usefulness of the tags.

[0307] Furthermore, the annotation means may refer to competitive information for the product and assign different tags to the purchase information depending on the competitive information when assigning the tags.

[0308] With this configuration, different tags can be assigned to a single purchase data item depending on the competitive information of the product (differences in the sales scenario of the product), thereby increasing the usefulness of the tags.

[0309] Furthermore, the system may include a processing execution means capable of performing a predetermined process using the tagged purchase information, wherein the annotation means is capable of generating annotation information in which multiple types of tags are attached to the one purchase information, and the processing execution means selects one tag from among the multiple types of tags included in the annotation information that is most suitable for the predetermined process, and performs the predetermined process using the purchase information to which that tag is attached.

[0310] This configuration enhances the usefulness of tags used in specific processes (such as product sales analysis and demand forecasting), and allows for effective utilization of product purchase information during the execution of these processes.

[0311] Furthermore, the processing execution means (processing execution step) according to Embodiment 3 is not an essential component of the information processing system, information processing method, and information processing program according to the present invention, and the system does not need to include the processing execution means (processing execution step).

[0312] Furthermore, in the "annotation means" according to Embodiment 3, when assigning multiple tags to a single piece of purchase information, each of the multiple tags may be weighted according to the importance of the analysis results of the management information. For example, the importance of tags assigned based on information that is highly important or frequently occurring in the management information may be set higher than the importance of other tags.

[0313] It should be noted that the information processing system, information processing method, and information processing program according to the present invention are not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the present invention.

[0314] Therefore, for example, at least a part of the configuration of the invention according to any of Embodiments 1 to 3 may be combined with at least a part of the configuration of the other invention according to any of Embodiments 1 to 3, or at least a part of the configuration of the invention according to any of Embodiments 1 to 3 may be replaced with the configuration of the other invention according to any of Embodiments 1 to 3. [Industrial applicability]

[0315] The information processing system, information processing method, and information processing program according to the present invention can be widely applied in manufacturing, service, retail, and other industries. [Explanation of symbols]

[0316] 10. Information Processing Systems 12 System Terminals 12a Acquisition method 12b Verification means 12c Substitution means 12d Annotation means 14 Information storage terminals 16 External terminals 21 CPU 22 ROM 23 RAM 24 Recording media 25 External storage drives 26 Storage device 27 Input devices 28 indicates the device 29 Ministry of Communications

Claims

1. An information processing system comprising a memory means and an AI agent, The storage means is capable of storing multimodal data, including multiple modality data relating to the same object. The aforementioned AI agent, An acquisition means capable of obtaining characteristic information from the modality data, A context determination means capable of determining the processing context when assigning tags to the aforementioned feature information, The annotation means is capable of assigning different tags to the same feature information according to the processing context determined by the context determination means, The processing context includes information about the purpose of the processing to be performed after tagging the feature information. An information processing system characterized by the following:

2. In the information processing system described in claim 1, The processing context further includes at least one of the following: information regarding when the data corresponding to the feature information was acquired, and information regarding where the data corresponding to the feature information was acquired. An information processing system characterized by the following:

3. In the information processing system according to claim 1 or 2, The annotation means is capable of generating annotation information in which multiple tags corresponding to multiple processing contexts are assigned to the same feature information. The AI ​​agent includes processing execution means capable of selecting a tag from among the plurality of tags included in the annotation information according to the processing context corresponding to the purpose of the processing to be performed, and executing the processing using the feature information to which the selected tag is attached. An information processing system characterized by the following:

4. In the information processing system according to claim 1 or 2, The AI ​​agent includes a matching means capable of comparing multiple feature information obtained from at least two modality data and detecting inconsistencies in the multiple feature information, The system includes a replacement means capable of performing a replacement process in which, when the aforementioned inconsistency is detected, one of the plurality of feature information is determined to be the correct feature information (hereinafter referred to as "correct feature information") based on at least one of the following: majority vote, confidence score, data acquisition time, and priority of information sources, and feature information different from the correct feature information (hereinafter referred to as "incorrect feature information") is replaced with the correct feature information. The annotation means is capable of tagging the modality data using the positive feature information. An information processing system characterized by the following:

5. In the information processing system described in claim 4, The plurality of feature information that the matching means matches are feature information obtained from modality data of different types. An information processing system characterized by the following:

6. In the information processing system described in claim 4, The system includes a means for assigning an inconsistency score according to the degree of inconsistency, The replacement means replaces the erroneous feature information with the correct feature information only when the mismatch score exceeds a predetermined threshold. An information processing system characterized by the following:

7. In the information processing system described in claim 4, The matching means is capable of matching multiple feature information obtained from at least three modality data and detecting inconsistencies in the multiple feature information. The replacement means is capable of performing a replacement process in which, when the inconsistency is detected, the correct feature information is determined by majority vote from among the plurality of feature information, and the incorrect feature information is replaced with the correct feature information. An information processing system characterized by the following:

8. In the information processing system described in claim 4, The system includes an evaluation means for evaluating the aforementioned positive feature information, The annotation means tags the modality data using the positive feature information only if the positive feature information meets predetermined criteria. An information processing system characterized by the following:

9. In the information processing system described in claim 4, The replacement means determines the positive feature information from among multiple modality data corresponding to multiple feature information for which inconsistencies have been detected, based on one of the confidence score, data acquisition time, and information source priority, and replaces the other feature information with the positive feature information. An information processing system characterized by the following:

10. In the information processing system described in claim 9, The AI ​​agent updates the confidence score corresponding to each modality data, or the priority of the information source corresponding to the modality data, based on the history of matching results of the modality data. An information processing system characterized by the following:

11. In the information processing system described in claim 4, The modality data includes data relating to the business information of the product seller and data relating to product purchase information. The system includes an information analysis means capable of analyzing the aforementioned management information, The annotation means can assign different tags to a single piece of purchase information according to the results of the analysis of the management information when assigning tags. An information processing system characterized by the following:

12. An information processing method performed by an information processing system configured to have memory means and an AI agent, The storage means is capable of storing multimodal data, including multiple modality data relating to the same object. The aforementioned AI agent, An acquisition step that allows for the acquisition of feature information from the aforementioned modality data, A context determination step capable of determining the processing context when assigning tags to the aforementioned feature information, Depending on the processing context determined by the context determination step, at least the annotation step can be performed, which can assign different tags to the same feature information. The processing context includes information about the purpose of the processing to be performed after tagging the feature information. An information processing method characterized by the following:

13. A program for an information processing system comprising a memory means and an AI agent, The storage means is capable of storing multimodal data, including multiple modality data relating to the same object. The aforementioned AI agent, An acquisition means capable of obtaining characteristic information from the modality data, A context determination means capable of determining the processing context when assigning tags to the aforementioned feature information, Depending on the processing context determined by the context determination means, it functions as an annotation means capable of assigning different tags to the same feature information. The processing context includes information about the purpose of the processing to be performed after tagging the feature information. An information processing program characterized by the following features.

Images