Devices, data structures, and computer-implemented methods for processing digital content.
The method uses Data-to-Text models and text-data similarity metrics to analyze dataset differences, facilitating anomaly detection and model retraining by identifying key concepts, thus addressing the challenges of interpreting synthetic and real-world data discrepancies in machine learning.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-09-05
- Publication Date
- 2026-03-18
AI Technical Summary
Existing machine learning workflows face challenges in interpreting and managing differences between synthetic and real-world data, as well as correctly predicted and incorrectly predicted data, which are critical for understanding domain shifts during deployment.
A computer-implemented method using a Data-to-Text model to generate element-level descriptions for datasets, followed by text-data similarity metrics to identify common concepts and rank them based on average similarity, enabling the detection of anomalies and facilitating model retraining through a CLIP-based search filter.
Enables interpretable and manageable understanding of dataset differences, allowing for anomaly detection and improved model performance by identifying key concepts that explain discrepancies between datasets.
Smart Images
Figure 2026049712000044 
Figure 2026049712000045 
Figure 2026049712000046
Abstract
Description
[Technical Field]
[0001] background This description relates to devices, data structures, and computer-implemented methods for processing digital content. [Background technology]
[0002] Understanding the differences between two datasets is a critical challenge in machine learning workflows, for example, when (i) comparing synthetic data with real-world data, (ii) comparing data correctly predicted and incorrectly predicted by a machine learning model, or (iii) understanding the domain shift from the model's training data to the data observed after deployment. Ideally, these differences should be described in natural language in a way that is interpretable and manageable. [Overview of the project] [Problems that the invention aims to solve]
[0003] Disclosure of the invention The multimodal platform model can process data modalities such as digital images and audio signals, and express semantics for data analysis, for example, in natural language.
[0004] For example, the semantic or geometric properties of data such as digital images and audio signals can be represented in natural language. Large-scale language models can process natural language and, in particular, can perform operations on natural language, such as summarization. [Means for solving the problem]
[0005] A computer-implemented method for processing digital content includes providing a first dataset containing elements, and providing a second dataset containing elements, wherein the digital content of each element of the dataset includes a digital image which is, for example, a video image, a radar image, a LiDAR image, an ultrasound image, a motion image, or a thermal image, or the content of each element of the dataset includes a digital audio signal, and the method further includes generating a first set of descriptions, particularly using a Data-to-Text model, wherein the first set of descriptions includes element-level descriptions of the elements of the first dataset, and the description of each element of the first dataset is determined according to the content of each element of the first dataset, and generating a second set of descriptions, particularly using a Data-to-Text model, wherein the second set of descriptions includes element-level descriptions of the elements of the second dataset, and the description of each element of the second dataset is determined according to the content of each element of the second dataset, and particularly large-scale language modeling Using a text-data similarity metric, determine common concepts present in the first dataset but not in the second dataset, or common concepts that appear less frequently in the second dataset than in the first dataset; determine a first set of multiple text-data similarities for elements of the first dataset, particularly using a text-data similarity metric, wherein the first set of multiple text-data similarities includes element-level and common-concept-level text-data similarity for pairs of content of one element in the first dataset and one common concept; determine a second set of multiple text-data similarities for elements of the second dataset, particularly using a text-data similarity metric, wherein the second set of multiple text-data similarities includes element-level and common-concept-level text-data similarity for pairs of content of one element in the second dataset and one common concept; and determine, for the first set of multiple text-data similarities, the average text-data similarity associated with each common concept according to the first set of multiple text-data similarities, on a common-concept basis.The method includes: determining an average text-data similarity for each common concept according to the second set of text-data similarities for each common concept; associating a rank with each common concept, wherein the rank is determined by the average text-data similarity associated with the common concept according to the first set of text-data similarities and the average text-data similarity associated with the common concept according to the second set of text-data similarities; selecting at least one common concept according to the rank associated with the common concept; and outputting the selected at least one common concept.
[0006] A common concept is text, particularly natural language text, that provides a hypothesis about the differences between the first and second datasets. Text-data similarity compares the similarity between the content of each element, i.e., a digital image or audio signal, and the text of each common concept. Text-data similarity quantifies the extent to which the content of the elements in a pair supports the evidence for the hypothesis provided by the common concept in the pair regarding the differences between the first and second datasets.
[0007] The hypothesis can be used to detect anomalies in a technical system by calculating the difference between a recent set of measurements in which the model makes errors (the first dataset) and a correctly classified reference dataset (the second dataset). The first dataset can be considered as anomalous or rare data, and at least one common concept can explain the difference, i.e., the core characteristics of the anomaly.
[0008] Determining a rank may involve ranking common concepts with a higher average text-data similarity across a first set of text-data similarity metrics higher than common concepts with a lower text-data similarity metric according to that first set of text-data similarity metrics.
[0009] Determining a rank may include ranking a common concept having a lower average text-data similarity among a second plurality of text-data similarities higher than a common concept having a higher text-data similarity according to the second plurality of text-data similarities.
[0010] The method may include obtaining the content of an element using a sensor, particularly, obtaining a digital image using a camera, obtaining a video image using a camera, obtaining a radar image using a radar sensor, obtaining a LiDAR image using a LiDAR sensor, obtaining an ultrasonic image using an ultrasonic sensor, obtaining a motion image using a motion sensor, or obtaining a thermal image using a thermal image sensor, or obtaining an audio signal using a microphone.
[0011] Particularly, when comparing synthetically generated data with real-world data, the content of an element of the first data set is the synthetically generated content, and the content of an element of the second data set is the content obtained using a sensor in the real world.
[0012] The method may include transmitting at least one common concept to at least one technical system, particularly, a test bench or a vehicle or a robot, in order to select the obtained content according to the at least one common concept. The method may interact with the technical system, for example, in the following manner. That is, the technical system collects data that causes the model to exhibit undesirable behavior, such as misclassification, and data for which the model behaves normally. The method explains the difference. Based on this explanation, new data can be collected from the data collected by the technical system so that problematic conditions are better guaranteed.
[0013] The method may include receiving the content of elements of a first dataset and / or a second dataset from at least one technical system, in particular, a testbench or a vehicle or robot. For example, a text description of at least one common concept can be used to send a CLIP-based search filter to a group of vehicles to select appropriate data that matches the text description. Based on this collected data, the model can be retrained.
[0014] The apparatus for processing digital content comprises at least one processor and at least one memory, the at least one memory containing instructions that are executable by the at least one processor and cause the apparatus to perform a method when executed by the at least one processor.
[0015] A computer program containing computer-readable instructions for causing the computer to perform the method may be provided when the method is to be executed by a computer.
[0016] A data structure comprising at least one data field for a first dataset containing elements, the data structure comprising at least one data field for a second dataset containing elements, where the digital content of each element of the elements includes a digital image which is, for example, a video image, a radar image, a LiDAR image, an ultrasound image, a motion image or a thermal image, or the content of each element of the elements includes a digital audio signal, the data structure comprising at least one data field for a first set of descriptions generated in particular using a Data-to-Text model, where the first set of descriptions includes element-level descriptions of the elements of the first dataset, where the description of each element of the first dataset is determined according to the content of each element of the first dataset, the data structure comprising at least one data field for a second set of descriptions generated in particular using a Data-to-Text model, where the second set of descriptions includes element-level descriptions of the elements of the second dataset, where the description of each element of the second dataset is determined according to the content of each element of the second dataset The data structure is determined according to the content of each element of the dataset, and comprises at least one data field for common concepts that are present in the first dataset but not in the second dataset, or common concepts that are less frequent in the second dataset than in the first dataset, in particular for common concepts determined by a large-scale language model, and comprises at least one data field for a first set of multiple text-data similarities determined for elements of the first dataset, in particular using a text-data similarity metric, and the first set of multiple text-data similarities includes element-level and common-concept-level text-data similarity for pairs of content of one element and one common concept in the first dataset, and comprises at least one data field for a second set of multiple text-data similarities determined for elements of the second dataset, in particular using a text-data similarity metric, and the second set of multiple text-data similarities includes element-level and common-concept-level text-data similarity for pairs of content of one element and one common concept in the second dataset,The data structure may be provided having at least one data field for the average text-data similarity associated with each common concept according to the first multiple text-data similarity, determined for a first multiple text-data similarity unit; the data structure may be provided having at least one data field for the average text-data similarity associated with each common concept according to the second multiple text-data similarity, determined for a second multiple text-data similarity unit; the data structure may be provided having at least one data field for the rank associated with the common concept in a common concept unit, where the rank is determined by the average text-data similarity associated with the common concept according to the first multiple text-data similarity and the average text-data similarity associated with the common concept according to the second multiple text-data similarity; and the data structure may be provided having at least one data field for at least one common concept selected according to the rank associated with the common concept.
[0017] Further exemplary embodiments are derived from the following description and drawings. [Brief explanation of the drawing]
[0018] [Figure 1] This is a schematic diagram showing equipment for processing digital content. [Figure 2] This is a flowchart that includes the steps of a method for processing digital content. [Figure 3] This is a schematic diagram showing the data structure. [Modes for carrying out the invention]
[0019] Figure 1 schematically shows a device 100 for processing digital content. The device 100 comprises at least one processor 102 and at least one memory 104. The device 100 includes, for example, an interface 106 to a technical system 110. The interface 106 is configured to receive digital content from the technical system. The interface 106 is configured to transmit at least one common concept to the technical system 110.
[0020] The technical system 110 is configured, for example, to select digital content according to at least one common concept and to transmit the selected digital content to the interface 106.
[0021] The technical system 110 may be a test bench, a vehicle, or a robot.
[0022] Digital content includes, for example, digital images or digital audio signals.
[0023] Digital images include, for example, video images, radar images, LiDAR images, ultrasound images, motion images, or thermal images.
[0024] The technical system 110 is configured to acquire content using, for example, a sensor 112. Instead of the sensor 112 located within the technical system 110, the device 100 may be equipped with a sensor 112.
[0025] Sensor 112 includes, for example, a camera for acquiring digital images or video images. Sensor 112 includes, for example, a radar sensor for acquiring radar images. Sensor 112 includes, for example, a LiDAR sensor for acquiring LiDAR images. Sensor 112 includes, for example, an ultrasonic sensor for acquiring ultrasonic images. Sensor 112 includes, for example, a motion sensor for acquiring motion images. Sensor 112 includes, for example, a thermal imaging sensor for acquiring thermal images. Sensor 112 includes, for example, a microphone for acquiring audio signals.
[0026] At least one memory 104 includes instructions that are executable by at least one processor 102 and cause the device 100 to perform a method for processing digital content when executed by at least one processor 102.
[0027] Figure 2 shows a flowchart that includes the steps of a method for processing digital content.
[0028] This method includes step 202.
[0029] Step 202 is the first dataset.
number
[0030] The first dataset has n elements
number
[0031] element
number
[0032] When evaluating real-world content, elements
number
[0033] The real-world content may be received from the technology system 110 or the sensor 112.
[0034] When evaluating content generated by synthesis, the elements
number
[0035] This method includes step 204.
[0036] Step 204 is the second dataset.
number
[0037] The second dataset has m elements.
number
[0038] element
number
[0039] When evaluating real-world content, elements
number
[0040] The real-world content may be received from the technology system 110 or the sensor 112.
[0041] When evaluating content generated by synthesis, the elements
number
[0042] element
number
[0043] Digital images include, for example, video images, radar images, LiDAR images, ultrasound images, motion images, or thermal images.
[0044] This method is not limited to processing digital content, including digital images.
number
[0045] For example, element
number
[0046] This method includes step 206.
[0047] Step 206 is the first set of descriptions
number
[0048] Set of descriptions C A This is determined, for example, using a Data-to-Text model f. Examples of Data-to-Text models f include BLIP2 (arXiv:2301.12597) or LLaVa (arXiv:2304.08485).
[0049] Set 1 C A This is the first dataset D A elements
number
number
number
number
number
number
number
[0050] This method includes step 208.
[0051] Step 208 involves generating a second set of descriptions
Number
[0052] The second set of descriptions C B is determined, for example, using the Data-to-Text model f.
[0053] The second set C B is of the elements of the second dataset D B
Number
Number
Number
Number
Number
Number
Number
Number
[0055] Step 210 is the first dataset D A The second dataset D exists in B A common concept that does not exist in the first dataset D A The second dataset D is better than the one in the first dataset. B This includes determining less frequent common concepts in the context.
[0056] Common concepts are determined, for example, using large-scale language models such as Mistral-7B (arXiv:2310.06825).
[0057] For example, the following steps are repeated N times (j=0, ..., N-1). a. C A From K descriptions, C B K descriptions are sampled uniformly and randomly from the data. b. Construct a first text prompt based on the sampled text description and prompt template. The prompt template may be, for example, as follows: "Two measurement sets D" A and D B The description
number
number
[0058] H j This involves L hypotheses h regarding the difference between two sets of measurements. j,l As a list, H j ={h j,1 ,…,h j,L} It can be interpreted as follows.
[0059] N reduction list H j,j=1,…,N-1 This is the first dataset D A The second dataset D exists in B A common concept that does not exist in the first dataset D A The second dataset D is better than the one in the first dataset. B It can be used as a less frequent common concept in [the context].
[0060] N reduction list H j,j=1,…,N-1 It may have redundancy.
[0061] To eliminate redundancy in the hypothesis, after repeating steps a, b, and c N times, the method may include generating a second text prompt as follows: "Bulleted list {h 1,1 ,…,h N-1,L} contains measurement set D A It exists but D B This list includes related concepts that are not present in the original. The bulleted list above is excessively verbose and detailed; it needs to be more concise while preserving the diversity of the concepts covered. Do not make the bullet points any longer or more detailed. Instead, abstract some concepts into more general ones. Note that verbose entries may be described in slightly different ways. Interpret verbosity as concepts that are "semantically similar." The list can be significantly shortened by leaving only one representative entry for groups of verbose entries. Do not remove entries that are not adequately represented by other entries. This method is not limited to this second text prompt. More advanced or less advanced prompts are possible and compatible.
[0062] A second text prompt is provided to the large-scale language model, and the model's response is used in the first dataset D. A The second dataset D exists in B A common concept that does not exist in the first dataset D A The second dataset D is better than the one in the first dataset. B Low-frequency common concepts in H = {h1, ..., h R Record it as}.
[0063] This method includes step 212.
[0064] Step 212 is the first dataset D A elements
number
number
number
[0065] The first set of multiple text-to-data similarities is determined, for example, using a text-to-data similarity metric.
[0066] Common concept of pairs h j and elements
number
number
[0067] This method includes step 214.
[0068] Step 214 is to use the second dataset D B elements
number
number
number
[0069] A second set of text-to-data similarity metrics is determined, for example, using a text-to-data similarity metric.
[0070] Common concept of pairs h j and elements
number
number
[0071] This method includes step 216.
[0072] Step 216 involves determining, for each of the first multiple, the average text-data similarity associated with each common concept according to the first multiple, on a unit basis of the common concept.
[0073] This method includes step 218.
[0074] Step 218 involves determining, for a second plurality, the average text-data similarity associated with each common concept according to the second plurality, on a common concept basis.
[0075] This method includes step 220.
[0076] Step 220 involves associating common concepts with ranks in units of common concepts.
[0077] The rank is determined by the average text-data similarity associated with the common concept according to a first multiple and the average text-data similarity associated with the common concept according to a second multiple.
[0078] Determining a rank may involve ranking common concepts that have a higher average text-to-data similarity in the first plurality of entities higher than common concepts that have a lower text-to-data similarity according to the first plurality of entities.
[0079] Determining a rank may involve ranking common concepts with lower average text-to-data similarity in the second set of elements higher than common concepts with higher text-to-data similarity according to the second set of elements.
[0080] Rank is, for example, element
number
[0081] For example, element
number
[0082] This method includes step 222.
[0083] Step 222 involves selecting at least one common concept based on the rank associated with that common concept.
[0084] This method includes step 224.
[0085] Step 224 involves outputting at least one selected common concept.
[0086] Step 224 may include transmitting at least one common concept to the technical system 110 via interface 106. The technical system 110 may select digital content acquired by the technical system 110 in accordance with the at least one common concept in order to select appropriate digital content that matches a text description, and transmit the selected digital content to interface 106.
[0087] Step 224 may also include transmitting at least one common concept to a plurality of technical systems configured as described for technical system 110.
[0088] For example, the technical system is a vehicle in a vehicle group. A text description of at least one common concept is transmitted to the vehicle group. The vehicle is configured to apply a CLIP-based search filter to select appropriate digital content that matches the text description and to transmit the selected digital content to device 100. The vehicle applies a CLIP-based search filter to select appropriate digital content that matches the text description and to transmit the selected digital content.
[0089] This method is applicable to model training. The model can be trained using digital content of elements, for example, for classification or semantic segmentation.
[0090] Additional digital content for training can be collected by transmitting at least one common concept and receiving selected digital content. Based on the collected digital content, the model can be retrained, for example, in step 224.
[0091] Figure 3 schematically shows the data structure 300 for digital content processing.
[0092] The data structure is, • The first dataset containing the elements, • A second dataset containing the elements, Here, the digital content of each element includes, for example, a digital image such as a video image, radar image, LiDAR image, ultrasound image, motion image, or thermal image, or the content of each element includes a digital audio signal. In particular, a first set of descriptions generated using a Data-to-Text model, wherein the first set of descriptions includes element-level descriptions of elements in a first dataset, and the description of each element in the first dataset is determined according to the content of each element in the first dataset. In particular, a second set of descriptions generated using a Data-to-Text model, wherein the second set of descriptions includes element-level descriptions of elements in a second dataset, and the description of each element in the second dataset is determined according to the content of each element in the second dataset. Common concepts that exist in the first dataset but not in the second dataset, or common concepts that are less frequent in the second dataset than in the first dataset, especially common concepts determined by large-scale language models. • A first set of multiple text-data similarities determined for elements of a first dataset using a text-data similarity metric, the first set of multiple text-data similarities including element-level and common-concept-level text-data similarities for pairs of content of one element of the first dataset and one common concept, • A second set of multiple text-data similarities determined for elements of a second dataset using a text-data similarity metric, which includes element-level and common-concept-level text-data similarities for pairs of content and common concepts of one element of the second dataset. • The average text-data similarity associated with each common concept according to the first multiple text-data similarities, determined for each common concept unit for the first multiple text-data similarities. • The average text-data similarity associated with each common concept according to the second set of text-data similarities, determined for each common concept unit. • A rank associated with a common concept unit, determined by the average text-data similarity associated with the common concept according to a first set of text-data similarity values and the average text-data similarity associated with the common concept according to a second set of text-data similarity values. It comprises at least one data field 302 for [the data].
Claims
1. A computer-implemented method for processing digital content, The aforementioned method, To provide a first dataset containing the elements (202), To provide a second dataset containing the elements (204), Includes, The digital content of each of the elements includes digital images, such as video images, radar images, LiDAR images, ultrasound images, motion images, or thermal images, or the content of each of the elements includes digital audio signals. The aforementioned method, In particular, generating a first set of descriptions using a Data-to-Text model (206), wherein the first set includes element-level descriptions of the elements of the first dataset, and the description of each element of the first dataset is determined according to the content of each element of the first dataset. In particular, generating a second set of descriptions using the Data-to-Text model (208), wherein the second set includes element-level descriptions of the elements of the second dataset, and the description of each element of the second dataset is determined according to the content of each element of the second dataset. In particular, using a large-scale language model, determine common concepts that are present in the first dataset but not in the second dataset, or common concepts that are less frequent in the second dataset than in the first dataset (210), In particular, determining a first plurality of text-data similarities for the elements of the first dataset using a text-data similarity metric (212), wherein the first plurality includes text-data similarities for element-level and common-concept-level pairs of content of one element of the first dataset and one common-concept, In particular, determining a second plurality of text-data similarities for the elements of the second dataset using the text-data similarity metric (214), wherein the second plurality includes text-data similarities for element-level and common-concept-level pairs of content of one element of the second dataset and one common-concept, For the first plurality, the average text-to-data similarity associated with each common concept according to the first plurality is determined on a common concept basis (216), With respect to the second plurality, the average text-to-data similarity associated with each common concept according to the second plurality is determined on a unit basis of the common concept (218), The common concept is associated with a rank in units of the common concept (220), wherein the rank is determined by the average text-to-data similarity associated with the common concept according to the first plurality and the average text-to-data similarity associated with the common concept according to the second plurality, Selecting at least one common concept according to the rank associated with the aforementioned common concept (222), Outputting the selected at least one common concept (224), A method characterized by including
2. The method according to claim 1, wherein determining the rank (220) includes ranking common concepts having a higher average text-to-data similarity in the first plurality more highly than common concepts having a lower text-to-data similarity according to the first plurality.
3. The method according to claim 1 or 2, wherein determining the rank (220) includes ranking common concepts having a lower average text-to-data similarity in the second plurality higher than common concepts having a higher text-to-data similarity according to the second plurality.
4. The aforementioned method, Acquiring the content of the element using a sensor (202), in particular, acquiring the digital image using a camera, acquiring the video image using a camera, acquiring the radar image using a radar sensor, acquiring the LiDAR image using a LiDAR sensor, acquiring the ultrasonic image using an ultrasonic sensor, acquiring the motion image using a motion sensor, or acquiring the thermal image using a thermal image sensor, or acquiring the audio signal using a microphone. The method according to any one of claims 1 to 3, including
5. The method according to any one of claims 1 to 4, wherein the content of the elements in the first dataset is content generated by synthesis (202), and the content of the elements in the second dataset is content acquired in the real world using sensors (204).
6. The aforementioned method, Transmitting the at least one common concept to at least one technical system, in particular a test bench or a vehicle or robot, in order to select the acquired content according to the at least one common concept (224) The method according to any one of claims 1 to 5, including the method described in any one of claims 1 to 5.
7. The aforementioned method, Receiving the content of the elements of the first dataset and / or the second dataset from at least one technical system (110), in particular from a test bench or a vehicle or robot (202). The method according to any one of claims 1 to 6, including the method described in any one of claims 1 to 6.
8. A device (100) for processing digital content, The aforementioned device (100) At least one processor (102), At least one memory (104), Equipped with, The at least one memory (104) is executable by the at least one processor (102) and includes instructions causing the device (100) to perform the method according to any one of claims 1 to 7 when executed by the at least one processor (102), A device (100) characterized by the following.
9. A computer program characterized in that, when executed by a computer, it includes a computer-readable instruction causing the computer to perform the method according to any one of claims 1 to 7.
10. A data structure (300), The data structure (300) comprises at least one data field (302) for a first dataset containing elements, The data structure (300) comprises at least one data field (302) for a second dataset containing elements, The digital content of each of the elements includes, for example, a digital image such as a video image, radar image, LiDAR image, ultrasound image, motion image, or thermal image, or the content of each of the elements includes a digital audio signal. The data structure (300) comprises at least one data field (302) for a first set of descriptions generated using a Data-to-Text model, wherein the first set includes element-level descriptions of the elements of the first dataset, and the description of each element of the first dataset is determined according to the content of each element of the first dataset. The data structure (300) comprises at least one data field (302) for a second set of descriptions generated using the Data-to-Text model, the second set comprising element-level descriptions of the elements of the second dataset, the description of each element of the second dataset being determined according to the content of each element of the second dataset, The data structure (300) comprises at least one data field (302) for common concepts that are present in the first dataset but not in the second dataset, or common concepts that are less frequent in the second dataset than in the first dataset, in particular for common concepts determined using a large-scale language model. The data structure (300) comprises at least one data field (302) for a first plurality of text-data similarities determined for the elements of the first dataset, in particular using a text-data similarity metric, wherein the first plurality includes element-level and common-concept-level text-data similarities for pairs of content of one element of the first dataset and one common concept, The data structure (300) comprises at least one data field (302) for a second plurality of text-data similarities determined for the elements of the second dataset, in particular using the text-data similarity metric, wherein the second plurality includes element-level and common-concept-level text-data similarities for pairs of content of one element of the second dataset and one common concept, The data structure (300) comprises at least one data field (302) for average text-to-data similarity, which is determined for the first plurality of common conceptual units and associated with each common concept according to the first plurality, The data structure (300) comprises at least one data field (302) for average text-to-data similarity, which is determined for the second plurality of common conceptual units and associated with each common concept according to the second plurality, The data structure (300) comprises at least one data field (302) for a rank associated with the common concept in a common concept unit, the rank being determined by the average text-to-data similarity associated with the common concept according to a first plurality and the average text-to-data similarity associated with the common concept according to a second plurality. The data structure (300) comprises at least one data field (302) for at least one common concept selected according to the rank associated with the common concept. A data structure (300) characterized by the following.