Data processing system, data processing method, and program
The data processing system addresses the challenge of balancing flexibility and reliability in data similarity verification by integrating a trained model with a deterministic algorithm, ensuring accurate and efficient data similarity determination.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- ID HLDG CORP
- Filing Date
- 2025-12-11
- Publication Date
- 2026-06-03
AI Technical Summary
Existing data verification techniques using learning models struggle to balance flexibility and reliability in determining the presence or absence of similar data, often resulting in unreliable results.
A data processing system that combines a trained model for probabilistic comparison with a deterministic comparison algorithm, where the latter takes precedence for high similarity and complements the former for low similarity, using vectorization and hashing to enhance reliability and flexibility.
The system provides reliable and flexible verification of data similarity by prioritizing deterministic results for high similarity and integrating probabilistic results for low similarity, enhancing overall accuracy and user confidence in data verification.
Smart Images
Figure 2026091292000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a data processing system, a data processing method, and a program.
Background Art
[0002] Patent Document 1 discloses a technique for performing keyword search by using similarity determination by a learning model.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In a technique for searching for similar data, that is, a technique for verifying the presence or absence of similar data, a learning model may be used. Generally, by using a learning model to verify the presence or absence of similar data, the ambiguity of data is more tolerated, and the presence or absence of similar data can be verified more flexibly. On the other hand, generally, the reliability of the verification result of the presence or absence of similar data using a learning model tends to be relatively low. Therefore, a technique for verifying the presence or absence of similar data while achieving both flexibility and reliability is desired.
Means for Solving the Problems
[0005] The present disclosure can be realized in the following forms.
[0006] (1) According to one embodiment of the present disclosure, a data processing system is provided. This data processing system includes: a first storage unit that stores a data group containing one or more data; a second storage unit that stores the transformed data group containing one or more data transformed using a transformation algorithm for performing at least one of vectorization and hashing; an acquisition unit that acquires target data to be compared with the data group; a first comparison unit that generates a first comparison result relating to the comparison between the target data and the data group by inputting the target data to a trained model that has been trained to output a comparison result comparing the data group and the input data according to the input data; a transformation unit that transforms the target data using the transformation algorithm; a second comparison unit that generates a second comparison result relating to the comparison between the target data and the data group by comparing the transformed target data and the transformed data group using a predetermined deterministic comparison algorithm; and an output unit that outputs result information corresponding to the first comparison result and the second comparison result. The first comparison result includes first information regarding the presence or absence of data similar to the target data at a predetermined first similarity, and second information regarding the presence or absence of data similar to the target data at a second similarity lower than the first similarity. The second comparison result includes third information regarding the presence or absence of data similar to the target data at the first similarity, and fourth information regarding the presence or absence of data similar to the target data at the second similarity. If the first information and the third information do not correspond, the result information includes overwrite information obtained by overwriting the first information with the third information, and complementary information obtained by complementing the second information and the fourth information. In this configuration, the output results are such that, regarding the presence or absence of similar data with relatively high similarity, the second comparison result from a deterministic comparison algorithm takes precedence over the first comparison result from a trained model, and regarding the presence or absence of similar data with relatively low similarity, the first and second comparison results complement each other. Therefore, it is possible to flexibly determine the presence or absence of similar data with relatively low similarity while increasing the reliability of information regarding the presence or absence of similar data with relatively high similarity. As a result, users can use the result information to verify the presence or absence of similar data while maintaining both flexibility and reliability. (2) In the above configuration, the second similarity may be divided into a plurality of similarity categories, and the supplementary information may be divided and output according to the plurality of similarity categories. With this configuration, the user can more effectively verify the presence or absence of similar data with a relatively low similarity. This disclosure can be implemented in forms other than the data processing system described above, such as a data processing method, a data processing device, a program for implementing a data processing method, a non-temporary recording medium on which the program is recorded, or a program product. The program product may be provided, for example, as a recording medium on which the program is recorded, or as a program product that can be distributed via a network. [Brief explanation of the drawing]
[0007] [Figure 1] This is an explanatory diagram showing the general configuration of the data processing system. [Figure 2] This is a block diagram of a data processing device. [Figure 3] This is a conceptual diagram illustrating data processing in a data processing system. [Figure 4] This is a flowchart of data processing. [Figure 5] This is a diagram illustrating the results. [Figure 6] This is an explanatory diagram showing an example of visual information displayed based on the results. [Modes for carrying out the invention]
[0008] A. First Embodiment: Figure 1 is an explanatory diagram showing the schematic configuration of the data processing system 50 in the first embodiment. In this embodiment, the data processing system 50 is configured as a search system for searching for data DT similar to target data T. More specifically, the data processing system 50 is configured as a video data search system for searching for data DT similar to target data T as video data.
[0009] In this embodiment, the data processing system 50 comprises a data processing device 100, a database device 200, and a user terminal device 300.
[0010] The database device 200 comprises a first storage unit 210 and a second storage unit 220. The first storage unit 210 stores a data group DG. The data group DG includes one or more data DTs. In this embodiment, the data DT is video data including multiple image data and audio data. Metadata describing the data DT may be associated with the data DT. Metadata about the video data may include, for example, data representing the length (playback time) of the video, resolution, file format, frame rate, creation date and time, author information, data size, etc.
[0011] The second storage unit 220 stores the converted data group DGc. The converted data group DGc corresponds to the data group DG converted using a predetermined conversion algorithm. The converted data group DGc contains one or more converted data DTc. The converted data DTc corresponds to the data DT converted using a predetermined conversion algorithm.
[0012] The transformation algorithm is an algorithm for performing at least one of vectorization and hashing. In this embodiment, the transformation algorithm is a vectorization algorithm for performing vectorization. That is, the transformed data DTc is vectorized data DT, and includes various feature vectors such as image feature vectors, text vectors, word vectors, audio feature vectors, time series feature vectors, and video length feature vectors. The video length feature vector is, for example, a feature vector relating to the length of video playback time. Note that some of the feature vectors in the transformed data DTc may be based on metadata. For example, text vectors and word vectors may be vectorized strings based on audio data included in data DT, or they may be vectorized strings based on metadata about data DT. The number of dimensions of the transformed data DTc may be arbitrary, for example, 512 dimensions.
[0013] Various vectorization algorithms can be used. More specifically, a vectorization algorithm can be a combination of one or more algorithms for extracting various features from the data DT. For vectorizing image data, algorithms such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF) can be used. For vectorizing text data such as sentence vectors and word vectors, algorithms using Bag of Words (BoW), Term Frequency-Inverse Document Frequency (TD-IDF), and Histogram of Oriented Gradients (HOG) can be used. For vectorizing audio data, algorithms using Mel-Frequency Cepstral Coefficients (MFCC) can be used. The vectorization algorithm may also be implemented by a machine learning model that has been trained to vectorize the input data.
[0014] In this embodiment, the converted data group DGc is stored in the second storage unit 220 in a database DB configured as a vector database. The database DB may be configured, for example, as a relational database type vector database, or as a distributed ledger shared in a distributed network.
[0015] In other embodiments, at least some of the functions of the database device 200 may be incorporated into the data processing device 100. That is, the data processing device 100 may include a first storage unit 210, a second storage unit 220, and so on. Furthermore, the first storage unit 210 and the second storage unit 220 may be provided in, for example, separate computers or separate database devices.
[0016] The user terminal device 300 is used by a user. The user terminal device 300 comprises a user terminal control unit 301 and a display unit 310. In this embodiment, the user terminal control unit 301 is configured as a computer and comprises a processor 302, a memory 303 including ROM and RAM, and an input / output interface. More specifically, the user terminal device 300 is configured as, for example, a smartphone, various tablet terminals, or a personal computer. The processor 302 executes a program stored in the memory 303 to realize various functions, such as the function of transmitting target data T to the data processing device 100 and the function of receiving result information FR, described later, from the data processing device 100. The program 304 may be, for example, a dedicated application program usable by the user terminal control unit 301, or it may be a function extension program that extends the functions of a general-purpose application (e.g., a web browser) usable by the user terminal control unit 301. The display unit 310 is configured as a touch-operable liquid crystal display and also functions as an input device in the user terminal device 300.
[0017] Figure 2 is a block diagram of the data processing device 100. The data processing device 100 is composed of a computer comprising a processor 101, memory 102, input / output interface 103, and internal bus 104. The processor 101, memory 102, and input / output interface 103 are connected via the internal bus 104 to enable bidirectional communication. A communication device 105 is connected to the input / output interface 103. The communication device 105 can communicate with external devices such as a user terminal device 300 and a database device 200 via wired or wireless communication. The memory 102 includes RAM and ROM. Various information such as program PG1 and comparison model MD1 is stored in the memory 102. By executing program PG1, the processor 101 realizes various functions, including those of an acquisition unit 110, a first comparison unit 115, a conversion unit 120, a second comparison unit 125, and an output unit 130. Program PG1 includes a conversion program PG1A and a comparison program PG1B.
[0018] FIG. 3 is a conceptual diagram for explaining data processing in the data processing system 50. As shown in FIGS. 2 and 3, the acquisition unit 110 acquires target data T to be compared with the data group DG. In the present embodiment, the acquisition unit 110 acquires the target data T transmitted from the user terminal device 300. In other embodiments, the acquisition unit 110 may acquire, for example, the target data T stored in advance in the memory 102, or may acquire the target data T from an external computer or a recording medium.
[0019] The first comparison unit 115 shown in FIGS. 2 and 3 executes a first comparison process. The first comparison process is a process of generating a first comparison result CR1 by inputting the target data T acquired by the acquisition unit 110 into the comparison model MD1.
[0020] The first comparison result CR1 is a comparison result regarding the comparison between the target data T and the data group DG, and corresponds to the comparison result of comparing the target data T and the data group DG. The first comparison result CR1 includes first information and second information. The first information is information regarding the presence or absence of data DT that is similar to the target data T at a predetermined first similarity. The second information is information regarding the presence or absence of data DT that is similar to the target data T at a second similarity. The second similarity is a similarity lower than the first similarity. Hereinafter, the first information and the second information indicating that there is data DT satisfying the similarity condition are also referred to as information indicating "yes". Also, the first information and the second information indicating that there is no data DT satisfying the similarity condition are also referred to as information indicating "no". In the present embodiment, the first information and the second information indicating "yes" include the URI (Uniform Resource Identifier) of the data DT and the similarity of the data DT. The URI includes at least one of a URN (Uniform Resource Name) and a URL (Uniform Resource Locator). Also, the first information and the second information indicating "no" include information indicating that there is no data satisfying the similarity condition. The information indicating that there is no data satisfying the similarity condition may be, for example, error information.
[0021] The comparison model MD1 is a trained machine learning model that is trained to output a comparison result obtained by comparing the data group DG and the input data according to the input data to the comparison model MD1. The comparison model MD1 outputs the comparison result probabilistically or in a black-box manner according to the input data. In the present embodiment, the comparison model MD1 outputs the comparison result probabilistically and in a black-box manner according to the input data. Thus, in the first comparison process, by using the probabilistic or black-box comparison model MD1, the first comparison result CR1 becomes probabilistic or black-box. Here, the term "probabilistic" specifically means that even if the target data T and the data group DG are fixed, the output comparison result is not uniquely determined. As a result, the first comparison result CR1 has a tendency to be less reliable while allowing more data ambiguity as compared with the second comparison result CR2 described later.
[0022] Note that, as the machine learning model as the comparison model MD1, for example, a neural network, a random forest, a linear regression analysis, a linear discriminant, reinforcement learning, or a probability model can be used. The machine learning model may be a deep learning model such as a convolutional neural network (CNN). Further, as the machine learning model, a generative model such as a generative adversarial network (GAN), a variational autoencoder (VAE), or an autoregressive model may be used. Further, the machine learning model may be learned using various learning algorithms such as supervised learning, unsupervised learning, or reinforcement learning. When the comparison model MD1 is learned by supervised learning, a learning dataset including a plurality of labeled data is used. In this case, as the label attached to a certain data in the learning dataset, the similarity between the data and the data DT is used. Such similarity as a label can be calculated using, for example, the same algorithm as the comparison algorithm described later.
[0023] In this embodiment, the first and second similarity measures are similarities based on cosine similarity, and more specifically, they represent cosine similarity. In this embodiment, the first similarity measure is defined as a similarity measure high enough to indicate an agreement between the target data T and the data DT. The second similarity measure is defined as a similarity measure high enough to indicate that the target data T and the data DT do not agree, but are relatively similar. For example, the first similarity measure is defined as being between 0.9 and 1.0, and the second similarity measure is defined as being between 0.7 and less than 0.9. In this embodiment, data DT having the first similarity measure is also referred to as "matching data." In other embodiments, the first and second similarity measures may not be limited to cosine similarity, but may be similarities based on Euclidean distance, Manhattan distance, Dot product, or Jacquard coefficients, for example.
[0024] In this embodiment, the second similarity is divided into multiple similarity categories. More specifically, the second similarity is divided into a first category and a second category with a lower similarity than the first category. For example, the first category is defined as a category with a similarity of 0.8 or more and less than 0.9, and the second category is defined as a category with a similarity of 0.7 or more and less than 0.8. In other embodiments, the second similarity may be divided into three or more similarity categories. Also, similar to the second similarity, the first similarity may be divided into multiple similarity categories.
[0025] As shown in Figures 2 and 3, the conversion unit 120 generates converted target data Tc by converting the target data T using a conversion algorithm. In other words, in this embodiment, the converted target data Tc is the vectorized target data T. The conversion unit 120 is implemented by the conversion program PG1A. The conversion program PG1A is a program that enables a computer to implement the function of converting the target data T using a conversion algorithm.
[0026] The second comparison unit 125 shown in Figures 2 and 3 executes a second comparison process. The second comparison process outputs a second comparison result CR2 using a predetermined deterministic comparison algorithm. The second comparison result CR2 is the comparison result of comparing the converted target data Tc with the converted data group DGc, and is a comparison result concerning the comparison between the target data T and the data group DG. In other words, the second comparison process compares the target data T and the data group DG by comparing the converted target data Tc with the converted data group DGc. Furthermore, "deterministic" here means that if the converted target data Tc and the converted data group DGc are fixed, the comparison result is uniquely determined. The second comparison unit 125 is implemented by the comparison program PG1B. The comparison program PG1B is a program that enables the computer to implement the function of executing the second comparison process.
[0027] The second comparison result CR2 includes third information and fourth information. The third information is information regarding the presence or absence of data DT that is similar to the target data T in terms of first similarity. The fourth information is information regarding the presence or absence of data DT that is similar to the target data T in terms of second similarity. Hereafter, as with the first and second information, the third and fourth information indicating that there is data DT that satisfies the similarity conditions will also be referred to as "present" information. Also, the third and fourth information indicating that there is no data DT that satisfies the similarity conditions will also be referred to as "absent" information. In this embodiment, the third and fourth information indicating "present" includes the URI of data DT that satisfies the similarity conditions. Also, the third and fourth information indicating "absent" includes information indicating that there is no data DT that satisfies the similarity conditions.
[0028] In this embodiment, corresponding to the first and second similarity values representing cosine similarity, a comparison algorithm utilizing cosine similarity is used. As a result, the second comparison result CR2 represents the comparison result based on the cosine similarity between the transformed target data Tc and the transformed data DTc. In other embodiments, in addition to cosine similarity, or instead, algorithms utilizing, for example, Euclidean distance, Manhattan distance, dot product, or Jaccard coefficients may be used as the comparison algorithm. Furthermore, deterministic comparison algorithms may be implemented using non-probabilistic machine learning models, such as decision trees or support vector machines.
[0029] As shown in Figure 3, in this embodiment, in the second comparison process, the second comparison unit 125 sends a query QE to the database device 200 for extracting converted data DTc having a first similarity and converted data DTc having a second similarity. The database device 200 sends response information RE in response to the query QE sent from the data processing device 100. The response information RE may include converted data DTc having a first similarity and its similarity, and converted data DTc having a second similarity and its similarity, depending on the data extraction conditions expressed in the query QE. Based on the response information RE, the second comparison unit 125 generates a second comparison result CR2.
[0030] As shown in Figures 2 and 3, the output unit 130 outputs result information FR corresponding to the first comparison result CR1 and the second comparison result CR2. The result information FR includes first output information and second output information. The first output information is information corresponding to the first information and the third information. If the first information and the third information do not correspond, the first information is overwritten by the third information in the first output information. In this case, the first output information includes the third information but does not include the first information. The second output information is information corresponding to the second information and the fourth information. The second output information includes information in which the second information and the fourth information complement each other. As a result, if the first information and the third information do not correspond, the result information FR includes overwritten information in which the first information is overwritten by the third information, and complementary information in which the second information and the fourth information complement each other.
[0031] In this embodiment, the similarity of data DT in the supplementary information is divided and output according to the similarity categories described above. The information representing the similarity categories may be included in the second information, for example, or may be stored in advance in memory 102 or the like.
[0032] In this embodiment, the output unit 130 outputs result information FR to the user terminal device 300 via the communication device 105. The user terminal control unit 301 of the user terminal device 300 outputs visual information to the display unit 310 based on the result information FR output from the data processing device 100. As a result, the result information FR is provided to the user as visual information.
[0033] Figure 4 is a data processing flowchart for implementing the data processing method in this embodiment. The search process shown in Figure 4 is started by the processor 101 of the data processing device 100 when a predetermined operation is performed by a user via the user terminal device 300.
[0034] In step S100, the acquisition unit 110 acquires the target data T. In step S105, the first comparison unit 115 executes the first comparison process. More specifically, in step S105, the first comparison unit 115 inputs the target data T acquired in step S100 into the comparison model MD1, causing the comparison model MD1 to output the first comparison result CR1, thereby generating the first comparison result CR1.
[0035] In step S110, the conversion unit 120 converts the target data T acquired in step S100 using a conversion algorithm. As a result, the converted target data Tc is generated.
[0036] In step S115, the second comparison unit 125 performs a second comparison process. In step S115, the second comparison unit 125 uses a deterministic comparison algorithm to compare the transformed target data Tc generated in step S110 with the transformed data group DGc, thereby generating a second comparison result CR2.
[0037] In step S120, the output unit 130 outputs result information FR corresponding to the first comparison result CR1 generated in step S105 and the second comparison result CR2 generated in step S115.
[0038] Figure 5 is a diagram illustrating the result information FR in this embodiment. In Figure 5, each piece of information from the first to the fourth is indicated as either "Yes" or "No". In Figure 5, "Yes" means that the information is "Yes". For example, "Yes:DT1" in the first piece of information for pattern PT1 in Figure 5 indicates that there is data DT1 that has the first similarity. Also in Figure 5, the data name of data DT that satisfies the similarity condition is indicated in the format "Yes:Data Name". In Figure 5, "No" means that the information is "No". For example, "No" in the third piece of information for pattern PT1 in Figure 5 indicates that there is no data DT that has the first similarity. Also in Figure 5, the output information is represented in the format "First Output Information, Second Output Information".
[0039] For example, in pattern PT1 in Figure 5, the first piece of information is "Yes: DT1", while the third piece of information is "No". Therefore, in pattern PT1, the first piece of information and the third piece of information do not correspond to each other. As a result, in the first output information of result information Inf1 output by pattern PT1, the first piece of information representing "Yes: DT1" is overwritten by the third piece of information representing "No". Similarly, in pattern PT2, the first piece of information represents "No", while the third piece of information represents "Yes: DT3". As a result, in the first output information of result information Inf2 output by pattern PT2, the first piece of information representing "No" is overwritten by the third piece of information representing "Yes: DT3". Similarly, in pattern PT3, the first piece of information represents "Yes: DT1", while the third piece of information represents "Yes: DT3". As a result, in the first output information of result information Inf3 output by pattern PT3, the first piece of information representing "Yes: DT1" is overwritten by the third piece of information representing "Yes: DT3".
[0040] Furthermore, for example, the second output information in the result information Inf6 output for pattern PT6 includes information representing "Yes:DT2,DT4," which is a complementary combination of the second information representing "Yes:DT2" and the fourth information representing "Yes:DT4." In addition, in other patterns in Figure 5, the result information FR also includes information where the second and fourth information are complementary.
[0041] In the patterns PT4, PT5, and PT6 shown in Figure 5, the first and third pieces of information are both "Yes: DT1" and correspond to each other. When the first and third pieces of information correspond in this way, in the result information FR, either the first or third piece of information may be overwritten by the other, or the first and third pieces of information may complement each other.
[0042] Figure 6 is an explanatory diagram showing an example of visual information Vi displayed on the display screen Sc of the display unit 310 based on result information FR. In the example of Figure 6, the visual information Vi includes first visual information Vi1 displayed in the first display area SA1 and second visual information Vi2 displayed in the second display area SA2. The first display area SA1 and the second display area SA2 are different display areas on the display screen of the display unit 310. The first visual information Vi1 is visual information based on the first output information, that is, information regarding the presence or absence of data DT having a first similarity. In the example of Figure 6, the first visual information Vi1 is displayed in a column named "Matching Data". The first visual information Vi1 includes the "TITLE", "SCORE", and "URL" of each data DT having a first similarity. In Figure 6, "TITLE" and "SCORE" correspond to the name and similarity of the data DT, respectively. The second visual information Vi2 is visual information based on the second output information, that is, information regarding the presence or absence of data DT with a second similarity. In the example in Figure 6, Vi2a, which is information about the data DT of the first category within the second visual information Vi2, is displayed in a column named "Similar Data." Similarly, Vi2b, which is information about the data DT of the second category within the second visual information Vi2, is displayed in a column named "Related Data." In other words, in the example in Figure 6, the second visual information Vi2 is output categorized according to its similarity classification. Note that the visual information Vi is not limited to "TITLE," "SCORE," and "URI" as shown in Figure 6; any information, such as various metadata, may be displayed.
[0043] As described above, the data processing system 50 in this embodiment outputs result information FR such that, regarding the presence or absence of similar data with a relatively high degree of similarity, the second comparison result CR2 obtained by a deterministic comparison algorithm takes precedence over the first comparison result CR1 obtained by a trained model, and regarding the presence or absence of similar data with a relatively low degree of similarity, the first comparison result CR1 and the second comparison result CR2 complement each other. Therefore, it is possible to prevent the first comparison result CR1, which negates the second comparison result CR2, from being reflected in the result information FR regarding the presence or absence of similar data with a relatively high degree of similarity. More specifically, in this embodiment, for example, when the second comparison result CR2 indicates that "matching data exists," it is possible to prevent the first comparison result CR1, which indicates that "matching data does not exist," from being reflected in the result information FR. Also, when the second comparison result CR2 indicates that "matching data does not exist," it is possible to prevent the first comparison result CR1, which indicates that "matching data exists," from being reflected in the result information FR. On the other hand, regarding the presence or absence of similar data with relatively low similarity, even if, for example, the first comparison result CR1 is negative compared to the second comparison result CR2, the first comparison result CR1 is not immediately excluded but is reflected in the result information FR. Therefore, it is possible to flexibly verify the presence or absence of similar data with relatively low similarity while increasing the reliability of information regarding the presence or absence of similar data with relatively high similarity. As a result, users can verify the presence or absence of similar data while maintaining both flexibility and reliability by using the result information FR.
[0044] Furthermore, in this embodiment, the second similarity is divided into multiple similarity categories, and the supplementary information in the result information FR is divided and output according to the similarity category. Therefore, the user can more effectively verify the presence or absence of similar data with relatively low similarity. More specifically, for example, when the user performs an authenticity determination to determine whether the target data T is a forgery of any of the data DT in the data group DG, the user can prioritize checking the data DT belonging to the similarity category with a higher similarity among the data DT with a second similarity included in the result information FR. In this way, the user can more efficiently and effectively verify, for example, whether the target data T is a forgery that reuses most of the data DT while changing only a small part of it.
[0045] B. Other embodiments: (B1) In the above embodiment, the data processing system 50 was configured as a search system for searching for data DT similar to the target data T as video data, but it is not limited to this. Furthermore, the target data T and data DT are not limited to video data, but may be any type of data, such as text data, audio data, log data, metadata, or program data.
[0046] For example, the data processing system 50 may be used for security deception. More specifically, the data processing system 50 may be used, for example, to detect data alteration due to a cyberattack in a honeypot. In this case, for example, log data from the honeypot at the first timing is used as the target data T, and log data from the second timing prior to the first timing is used as the data DT. In this configuration, if the third information represents "present," the probability of data alteration occurring between the first and second timings is low. Also, if the third information represents "absent," the probability of data alteration occurring between the first and second timings is high. In particular, if the third information represents "absent," and the second and fourth information also represent "absent," the probability of a relatively large degree of data alteration between the first and second timings is high. Also, if the third information represents "absent," and the second and fourth information also represent "present," the probability of a relatively small degree of data alteration between the first and second timings is high. In this way, by checking the result information FR, users can confirm with high accuracy whether or not the data has been altered, and can verify whether or not a critical data alteration has occurred based on the degree of the data alteration. Furthermore, even if the third information indicates "yes," for example, if the second information indicates "yes," the result information FR includes the first output information reflecting the third information indicating "yes" and the second output information reflecting the second information indicating "yes." In this case, users can understand that the probability of data alteration is low, and can verify the possibility that an alteration of a relatively small degree has occurred. As a result, users can use the result information FR to verify more carefully whether or not data alteration has occurred.
[0047] (B2) In each of the above embodiments, the transformation algorithm may be any algorithm that performs at least one of vectorization and hashing, for example, a hashing algorithm for performing hashing. Alternatively, the transformation algorithm may be a transformation algorithm that uses a combination of vectorization and hashing. The hashing algorithm is preferably an algorithm that can generate a hash value that reflects the characteristics of the data before hashing, such as MinHash or SimHash. In this case, as a deterministic comparison algorithm, it is preferable to use, for example, a comparison algorithm that utilizes cosine similarity, Hamming distance, or Jacquard coefficients.
[0048] (B3) In each of the above embodiments, the result information FR is output to the user terminal device 300, but is not limited to this. For example, the result information FR may be output to various output devices and control devices connected to the data processing device 100 by wire or wireless. Output devices include, for example, display devices such as displays that show visual information, audio output devices such as speakers that output audio information, and printing devices.
[0049] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]
[0050] 50...Data processing system, 100...Data processing device, 101...Processor, 102...Memory, 103...Input / Output interface, 104...Internal bus, 105...Communication device, 110...Acquisition unit, 115...First comparison unit, 120...Conversion unit, 125...Second comparison unit, 130...Output unit, 200...Database device, 210...First storage unit, 220...Second storage unit, 300...User terminal device, 301...User terminal control unit, 302...Processor, 303...Memory, 304...Program, 310...Display unit
Claims
1. A first storage unit that stores a data set containing one or more data items, A second storage unit stores the transformed data set, which includes one or more data transformed using a transformation algorithm for performing at least one of vectorization and hashing. An acquisition unit that acquires target data to be compared with the aforementioned data group, A first comparison unit generates a first comparison result regarding the comparison between the target data and the data group by inputting the target data into a trained model that has been trained to output a comparison result by comparing the data group with the input data according to the input data. A conversion unit that converts the target data using the conversion algorithm, A second comparison unit generates a second comparison result regarding the comparison between the target data and the data group by comparing the converted target data and the converted data group using a predetermined deterministic comparison algorithm. The system includes an output unit that outputs result information corresponding to the first comparison result and the second comparison result, The first comparison result includes first information regarding the presence or absence of data similar to the target data at a predetermined first similarity level, and second information regarding the presence or absence of data similar to the target data at a second similarity level lower than the first similarity level. The second comparison result includes third information regarding the presence or absence of data similar to the target data in the first similarity, and fourth information regarding the presence or absence of data similar to the target data in the second similarity, A data processing system in which the result information includes, if the first information and the third information do not correspond, overwrite information obtained by overwriting the first information with the third information, and complementary information obtained by complementing the second information and the fourth information.
2. A data processing system according to claim 1, The aforementioned second similarity is divided into multiple similarity categories, A data processing system that outputs the aforementioned supplementary information, categorized according to the multiple similarity categories.
3. A step of acquiring target data to be compared with a data set containing one or more data points, A step of generating a first comparison result regarding the comparison between the target data and the data group by inputting the target data into a trained model that has been trained to output a comparison result of comparing the data group with the input data according to the input data, A step of transforming the target data using a transformation algorithm that performs at least one of vectorization and hashing, A step of generating a second comparison result regarding the comparison between the target data and the data group by comparing the converted target data with the converted data group, which includes the one or more data converted using the conversion algorithm, using a predetermined deterministic comparison algorithm. The process includes a step of outputting result information corresponding to the first comparison result and the second comparison result, The first comparison result includes first information regarding the presence or absence of data similar to the target data at a predetermined first similarity level, and second information regarding the presence or absence of data similar to the target data at a second similarity level lower than the first similarity level. The second comparison result includes third information regarding the presence or absence of data similar to the target data in the first similarity, and fourth information regarding the presence or absence of data similar to the target data in the second similarity, A data processing method in which the result information includes, if the first information and the third information do not correspond, overwrite information obtained by overwriting the first information with the third information, and complementary information obtained by complementing the second information and the fourth information.
4. A function to retrieve target data to be compared with a data set containing one or more data points, A function to generate a first comparison result regarding the comparison between the target data and the data set by inputting the target data into a trained model that has been trained to output a comparison result by comparing the data set with the input data according to the input data, A function to transform the target data using a transformation algorithm that performs at least one of vectorization and hashing, A function to generate a second comparison result regarding the comparison between the target data and the data group by comparing the converted target data with the converted data group, which includes the one or more data converted using the conversion algorithm, using a predetermined deterministic comparison algorithm. The computer is provided with a function to output result information corresponding to the first comparison result and the second comparison result. The first comparison result includes first information regarding the presence or absence of data similar to the target data at a predetermined first similarity level, and second information regarding the presence or absence of data similar to the target data at a second similarity level lower than the first similarity level. The second comparison result includes third information regarding the presence or absence of data similar to the target data in the first similarity, and fourth information regarding the presence or absence of data similar to the target data in the second similarity, The program includes, if the first information and the third information do not correspond, overwrite information obtained by overwriting the first information with the third information, and complementary information obtained by complementing the second information and the fourth information.