Method, device, electronic equipment, computer readable storage medium and computer program product for data processing

Abstract

The application provides a data processing method and device, electronic equipment, computer program product and computer readable storage medium; the method comprises: performing disassembly processing on a first vector of a first text to obtain a plurality of third vectors having the same dimension number as a second vector of a second text; performing numerical difference calculation processing on the second vector and each third vector to obtain a first vector difference between the second vector and each third vector; determining first element change data in the second vector based on a plurality of first element values included in the second vector, and determining second element change data in the third vector based on a plurality of second element values included in the third vector; determining a second vector difference between the second vector and each third vector based on the first element change data and the second element change data corresponding to each third vector; and performing fusion processing on the first vector difference and the second vector difference to obtain a text difference degree between the first text and the second text.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, more and more scenarios require the use of textual discriminant metrics, such as information retrieval and cluster analysis. Determining the textual discriminant metrics involves identifying the differences between two texts of different lengths. Since texts of different lengths have different dimensions for their text vectors, a common approach is to unify the dimensions of the two text vectors using an existing model. While this method can yield the textual discriminant metrics, the adjustment required during the unification process leads to inaccurate representations of the adjusted text vectors, resulting in inaccurate determined textual discriminant metrics. Summary of the Invention

[0003] This application provides a method, apparatus, computer-readable storage medium, and computer program product that can improve the accuracy of the determined text differences between texts.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, the method comprising:

[0006] The first vector of the first text is decomposed to obtain multiple third vectors with the same number of dimensions as the second vector of the second text;

[0007] The numerical difference between the second vector and each of the third vectors is calculated to obtain the first vector difference between the second vector and each of the third vectors.

[0008] Based on the multiple first element values ​​included in the second vector, determine the first element change data within the second vector, and based on the multiple second element values ​​included in the third vector, determine the second element change data within the third vector;

[0009] Based on the first element change data and the second element change data corresponding to each of the third vectors, the second vector difference between the second vector and each of the third vectors is determined;

[0010] The first vector difference and the second vector difference are fused to obtain the text difference degree between the first text and the second text.

[0011] This application provides a data processing apparatus, including:

[0012] The decomposition module is used to decompose the first vector of the first text to obtain multiple third vectors with the same number of dimensions as the second vector of the second text.

[0013] The calculation module is used to perform numerical difference calculation processing on the second vector and each of the third vectors to obtain the first vector difference between the second vector and each of the third vectors;

[0014] The determining module is used to determine the first element change data in the second vector based on the multiple first element values ​​included in the second vector, and to determine the second element change data in the third vector based on the multiple second element values ​​included in the third vector;

[0015] The determining module is further configured to determine the second vector difference between the second vector and each of the third vectors based on the first element change data and the second element change data corresponding to each of the third vectors;

[0016] The fusion module is used to fuse the first vector difference and the second vector difference to obtain the text difference degree between the first text and the second text.

[0017] This application provides an electronic device, the electronic device comprising:

[0018] Memory is used to store executable instructions or computer programs.

[0019] A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the data processing method provided in this application.

[0022] The embodiments of this application have the following beneficial effects:

[0023] The first vector of the first text is decomposed to obtain multiple third vectors with the same number of dimensions as the second vector of the second text. By decomposing the vectors, multiple sets of vectors with the same dimensions are obtained. Since the vectors themselves are not modified, the information contained in the vectors is preserved, thus ensuring the accuracy of the text difference degree determined subsequently between the texts. The numerical difference between the second vector and each of the third vectors is calculated to obtain the first vector difference between the second vector and each of the third vectors. Here, the first vector difference is equivalent to a numerical difference. Based on the multiple first element values ​​included in the second vector, the change data of the first element within the second vector is determined, and based on the multiple second element values ​​included in the third vector, the change data of the second element within the third vector is determined. Based on the change data of the first element and the change data of the second element corresponding to each of the third vectors, the second vector difference between the second vector and each of the third vectors is determined. Here, the second vector difference is equivalent to a change trend difference. The first vector difference and the second vector difference are fused to obtain the text difference degree between the first text and the second text. By combining the numerical difference between vectors and the change trend difference of the element values ​​within vectors, the text difference degree between the first text and the second text is determined, which improves the accuracy of the determined text difference degree. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the architecture of the data processing system 100 provided in the embodiments of this application;

[0025] Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0026] Figure 3 This is a first flowchart illustrating the data processing method provided in the embodiments of this application.

[0027] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0031] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0032] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0033] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0034] This application provides a data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of determining text differences between texts. The following describes exemplary applications of the data processing device provided in this application. The device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. The following will describe exemplary applications when the device is implemented as a server.

[0035] See Figure 1 , Figure 1This is a schematic diagram of the architecture of the data processing system 100 provided in the embodiments of this application. In order to support a voiceprint application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0036] Terminal 400 is used to acquire multiple texts from a business scenario and transmit the acquired texts to server 200. Server 200 is used to decompose the first vector of the first text to obtain multiple third vectors with the same dimension as the second vector. Then, the difference of the first vector is determined by the text difference degree between the second vector and the multiple third vectors. The difference of the second vector is determined by the element change data of the second vector and the element change data of the third vector. The difference of the first vector and the difference of the second vector are fused to obtain the text difference degree between the first text and the second text.

[0037] The data processing method provided in this application can be applied to scenarios where it is necessary to determine the degree of textual difference between texts, such as information retrieval scenarios and customer service scenarios.

[0038] The following example illustrates this:

[0039] 1) Information retrieval scenario: In the information retrieval scenario, terminal 400 can obtain the text to be retrieved input by the user, and then transmit the text to be retrieved to server 200. Server 200 trims the first vector of the text to be retrieved to obtain multiple third vectors with the same dimensions as the second vector of the text stored in the database. Then, the difference between the first vector is determined by the text difference degree between the second vector and the multiple third vectors, and the difference between the second vector is determined by the element change data of the second vector and the element change data of the third vector. The first vector difference and the second vector difference are fused to obtain the text difference degree between the text to be retrieved and the text stored in the database. Then, the text content that the user needs to retrieve is obtained based on the determined text difference degree.

[0040] 2) Customer Service Scenario: In the customer service scenario, terminal 400 can obtain the text of the user's inquiry, and then transmit the text of the user's inquiry to server 200. Server 200 trims the first vector of the inquiry text to obtain multiple third vectors with the same dimensions as the second vector of the text stored in the database. Then, it determines the first vector difference by the text difference degree between the second vector and the multiple third vectors, and determines the second vector difference by the element change data of the second vector and the element change data of the third vector. The first vector difference and the second vector difference are fused to obtain the text difference degree between the text to be retrieved and the text stored in the database. Then, based on the determined text difference degree, the answer to the user's inquiry is obtained and the answer is fed back to the user.

[0041] Terminal 400 can obtain the operation environment information and instruction information of the game interface and transmit them to server 200. Server 200 can decompose the user's instructions to be executed (such as creating a game room, inviting game friends, purchasing virtual characters, etc.) into at least one operation step, and then execute at least one operation step based on the operation environment information and instruction information of the game interface. After the operation step is completed, the consistency between the expected execution result and the actual execution result is checked.

[0042] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0043] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application, with the electronic device 500 as an example. Figure 1 Taking server 200 as an example, Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0044] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0045] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0046] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0047] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0048] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0049] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0050] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0051] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0052] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0053] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data processing apparatus 555, stored in memory 550, is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a decomposition module 5551, a calculation module 5552, a determination module 5553, and a fusion module 5554. These modules are logically connected and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0054] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0055] The data processing method provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the data processing method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0056] See Figure 3 , Figure 3 This is a first flowchart illustrating the data processing method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0057] In step 101, the first vector of the first text is decomposed to obtain multiple third vectors that have the same number of dimensions as the second vector of the second text.

[0058] In practice, the first text and the second text can be instant text sent by the user or historical text stored in the database. The difference between the first text and the second text is that the first text has more characters than the second text.

[0059] In some embodiments, before performing the decomposition process of the first vector of the first text in step 101 to obtain multiple third vectors with the same dimensions as the second vector of the second text, the following technical solution may also be performed: if the text length of the first text is greater than the text length threshold, the model corresponding to the number of first dimensions is called to perform vectorization processing on the first text to obtain a first vector with the number of first dimensions; if the text length of the second text is not greater than the text length threshold, the model corresponding to the number of second dimensions is called to perform vectorization processing on the second text to obtain a second vector with the number of second dimensions.

[0060] In practice, if the difference between the length of the first text and the length of the second text is too small, it may result in the first vector of the first text and the second vector of the second text having the same number of dimensions. For example, if the first text is "Is the weather nice today?" and the second text is "How is the weather?", then the difference between the lengths of the first text and the second text is only one, which may result in the first vector of the first text and the second vector of the second text having the same number of dimensions. In this case, it is not necessary to decompose the first vector of the first text. Therefore, it is necessary to select the first text and the second text that need to be decomposed.

[0061] In practice, for two texts whose text length needs to be determined, if the text length of one text is greater than the text length threshold and the text length of the other text is less than the text length threshold, then the text with the greater than the text length threshold can be taken as the first text and the text with the less than the text length threshold can be taken as the second text.

[0062] In practice, taking a customer service scenario as an example, when the user inputs the consultation text, the corresponding consultation questions are often filtered from the database. There are often multiple consultation questions that match the consultation text, and the text length of each consultation question may be different. Therefore, the average text length of the consultation questions stored in the database can be used as the text length of the text.

[0063] In practical implementation, after determining the first and second texts, pre-trained models corresponding to the first dimension and the second dimension can be called respectively to vectorize the first and second texts. The training methods for the first and second dimension models are as follows: First, collect a large amount of text data that represents the text types in the model's future application scenarios. For example, to build a text vector model for news text processing, various news articles need to be collected as training data, and the text needs to be segmented into individual words or sub-word units. For Chinese text, word segmentation tools such as Jieba can be used; for English text, simple word segmentation using spaces can be performed. Next, select a model structure, such as a continuous bag-of-words model or a skip-word model. Based on the selected model architecture, initialize the model's weights and biases, and choose an appropriate loss function based on the model's task and objective. For example, negative sampling loss function is commonly used in word embedding models; cross-entropy loss function is commonly used in text classification tasks. The training data is divided into mini-batches, and each mini-batch of data is input into the model for forward propagation to calculate the loss value. The gradient is calculated based on the loss function, and the gradient is passed back to the model's parameters through the backpropagation algorithm to calculate the gradient of each parameter. The optimizer is used to update the model's parameters based on the calculated gradients to minimize the loss function. The above process of batch training, backpropagation, and parameter update is repeated until the preset training rounds are completed.

[0064] It should be noted that the difference between training the model corresponding to the number of the first dimension and training the model corresponding to the number of the second dimension is that the training samples used when training the model corresponding to the number of the first dimension are texts with a length greater than the text length threshold, while the training samples used when training the model corresponding to the number of the second dimension are texts with a length less than the text length threshold.

[0065] By using the above method, the first and second texts that need to be decomposed can be selected without decomposing all the texts, thus saving computational resources. At the same time, by using different models to vectorize the first and second texts, the accuracy of obtaining the first vector of the first text and the second vector of the second text is improved.

[0066] In some embodiments, the process of decomposing the first vector of the first text in step 101 to obtain multiple third vectors with the same number of dimensions as the second vector of the second text can be achieved by the following technical solution: subtracting the number of the first dimension of the first vector from the number of the second dimension of the second vector, and adding one to the subtraction result to obtain a first number; decomposing the first vector to obtain the first number of third vectors with the second number of dimensions.

[0067] In practice, if the first dimension of the first vector has n elements and the second dimension of the second vector has m elements, then the total number of first dimensions is n-m+1, and the total number of third dimensions after decomposition is n-m+1. Specifically, if the first dimension of the first vector has 5 elements and the second dimension of the second vector has 3 elements, then the total number of third dimensions is 3.

[0068] In practice, the specific way to decompose the first vector is as follows: if the first vector is (1, 2, 3, 4, 5) (the number of the first dimension is 5) and the second vector is (1, 2, 3) (the number of the second dimension is 3), then starting from the starting position of the first vector, it can be decomposed in sequence according to the number of the second dimension to obtain the third vector A (1, 2, 3), the third vector B (2, 3, 4), and the third vector C (3, 4, 5).

[0069] By using the above method, the first vector can be decomposed into a third vector with the same number of dimensions as the second vector, and it is ensured that the information represented by the multiple third vectors is the same as the information represented by the first vector, thus ensuring the accuracy of the subsequent determination of the text difference between the first text and the second text.

[0070] In step 102, the numerical difference between the second vector and each of the third vectors is calculated to obtain the first vector difference between the second vector and each of the third vectors.

[0071] In some embodiments, the second vector includes N first element values, and the third vector includes N second element values, where N is a positive integer. Step 102, which calculates the numerical difference between the second vector and each of the third vectors to obtain the first vector difference between the second vector and each of the third vectors, can be achieved through the following technical solution: For each third vector, perform the following processing: determine the first element difference between the nth first element value in the second vector and the nth second element value in the third vector, where n ranges from 1 to N; perform a fusion process on the N first element differences to obtain a first fusion result; and determine the first vector difference between the second vector and the third vector based on the standard deviation of the second vector and the first fusion result.

[0072] In practical implementation, the method for determining the first vector difference can be found in the following formula (1):

[0073]

[0074] In formula (1), s j For the first vector difference, A i Let C be the value of the i-th first element in the second vector. j,iLet σ(A) be the value of the second element of the j-th third vector, and let σ(A) represent the standard deviation of the second vector; ∈ is a small positive number to avoid a denominator of 0, |A i -C j,i | represents the difference in the first element.

[0075] In the above formula (1), |A i -C j,i | represents the first element value of the second vector and the third vector C. j The absolute value of the difference between the i-th second element values ​​measures the distance between the second vector and the third vector in the i-th spatial dimension; This measures the relative deviation of the i-th element of the third vector from the second vector. Dividing the denominator by σ(A) is to ensure that the relative deviation is not excessively affected by internal fluctuations within the second vector, thus allowing for a fair comparison between different vectors; s j It measures the degree of deviation between the second and third vectors in N-dimensional space. The larger the value, the smaller the textual difference between the two vectors.

[0076] The above method can improve the accuracy of the determined first vector difference.

[0077] In step 103, based on the multiple first element values ​​included in the second vector, the change data of the first element in the second vector is determined, and based on the multiple second element values ​​included in the third vector, the change data of the second element in the third vector is determined.

[0078] In some embodiments, the second vector includes N first element values, where N is a positive integer. Step 103, based on the multiple first element values ​​included in the second vector, determines the first element change data within the second vector through the following technical solution: determining the second element difference between the (m+1)th first element value and the mth first element value in the second vector, where m ranges from 1 to N-1; determining the first ratio between the second element difference and the first maximum value as the first element change data corresponding to the mth first element value, wherein the first maximum value is the larger value between the (m+1)th first element value and the mth first element value; and forming the first element change data within the second vector from the N-1 first element values.

[0079] In practical implementation, the method for determining the change data of the first element can be found in the following formula (2):

[0080]

[0081] In formula (2), A jFor the change data of the first element corresponding to the i-th first element value, A i+1 Let A be the (i+1)th first element value in the second vector (i.e., the (m+1)th first element value). i Let |A be the value of the i-th first element in the second vector (i.e., the m-th first element). i+1 -A i | represents the difference of the second element, max(A) i+1 A i ) is the first maximum value. This is the first ratio. In the above formula (2), |A i+1 -A i | represents the absolute value of the difference between the (i+1)th and ith first elements of the second vector, measuring the magnitude between adjacent elements, max(A i+1 A i This means taking the maximum value between the (i+1)th first element of the second vector and the ith first element of the second vector, with the aim of finding the maximum value between |A| and |A|. i+1 -A i Normalization.

[0082] The above methods can improve the accuracy of the determined changes in the first element.

[0083] In some embodiments, the third vector includes N second element values, where N is a positive integer. Step 103, determining the second element change data within the third vector based on the multiple second element values ​​included in the third vector, can be achieved through the following technical solution: For each third vector, perform the following processing: determine the third element difference between the (m+1)th second element value and the mth second element value in the third vector, where m ranges from 1 to N-1; determine the second ratio between the third element difference and the second maximum value as the second element change data corresponding to the mth second element value, where the second maximum value is the larger value between the (m+1)th second element value and the mth second element value; and combine the second element change data corresponding to the N-1 second element values ​​to form the second element change data within the third vector.

[0084] In actual implementation, the method for determining the change data of the second element can be found in the following formula (3):

[0085]

[0086] In formula (3), C j For the change data of the second element corresponding to the i-th second element value, C j,i+1 In the j-th third vector, the (i+1)-th second element value (i.e., the (m+1)-th second element value), C j,iFor the j-th third vector, the i-th second element value (i.e., the m-th second element value), |C j,i+1 -C j,i | represents the difference of the third element, max(C) j,i+1 C j,i ) is the second maximum value. For the second ratio, in the above formula (3), |C j,i+1 -C j,i | represents the absolute value of the difference between the (i+1)th and ith second elements of the j-th third vector, measuring the magnitude between adjacent elements, max(C j,i+1 C j,i This means taking the maximum value between the (i+1)th and ith second elements of the j-th third vector, with the aim of finding the maximum value between |C|. j,i+1 -C j,i Normalization.

[0087] The above methods can improve the accuracy of the determined changes in the second element.

[0088] In step 104, based on the first element change data and the second element change data corresponding to each of the third vectors, the second vector difference between the second vector and each of the third vectors is determined.

[0089] In some embodiments, the second element change data within the third vector includes second element change data corresponding to N-1 second element values, and the first element change data within the second vector includes first element change data corresponding to N-1 first element values. The step 104, based on the first element change data and the second element change data corresponding to each third vector, determines the second vector difference between the second vector and each third vector. This can be achieved through the following technical solution: For each third vector, perform the following processing: determine the second element difference between the first element change data corresponding to the m-th first element value of the second vector and the second element change data corresponding to the m-th second element value of the third vector, where m ranges from 1 to N-1; perform fusion processing on the N-1 second element differences to obtain the second vector difference between the second vector and the third vector.

[0090] In practical implementation, the method for determining the difference of the second vector can be found in the following formula (4):

[0091]

[0092] In formula (4), d j Let the second vector and the third vector C be... j The second vector difference between them, Ai+1 Let A be the (i+1)th first element value in the second vector, where i is the number of first element values. i Let |A| be the value of the first element of the i-th vector. i+1 -A i | represents the difference of the second element, max(A) i+1 A i ) is the first maximum value. As the first ratio, C j,i+1 The j-th third vector contains the (i+1)-th second element value, where i is the number of second element values, and C j,i Let |C be the value of the second element of the j-th third vector, |C j,i+1 -C j,i | represents the difference of the third element, max(C) j,i+1 C j,i ) is the second maximum value. The second ratio is given. In the above formula (4), the difference between each first element difference and the second element difference is summed, and the absolute value of the summation is averaged to obtain the second vector difference. The second vector difference represents the average value of the changing trend of the second vector and the third vector in N-dimensional space. The larger the value of the second vector difference, the smaller the textual difference between the two vectors.

[0093] The above method can improve the accuracy of the determined second vector difference.

[0094] In step 105, the first vector difference and the second vector difference are fused to obtain the text difference degree between the first text and the second text.

[0095] In some embodiments, the fusion processing of the first vector difference and the second vector difference in step 105 to obtain the text difference degree between the first text and the second text can be achieved by the following technical solution: determining the first weight of the first vector difference and the second weight of the second vector difference; performing the following processing for each third vector: based on the first weight and the second weight, fusing the first vector difference between the second vector and the third vector and the second vector difference between the second vector and the third vector to obtain the vector distance between the second vector and the third vector; and taking the maximum text difference degree among the vector distances corresponding to the multiple third vectors as the text difference degree between the first text and the second text.

[0096] In practice, the method for determining the textual differences between the first and second texts can be found in the following formula (5):

[0097] S(A,B)=max(α*d j +(1-α)*exp(s j (5)

[0098] In formula (5), S(A,B) represents the text difference between the first and second texts, α is the first weight, (1-α) is the second weight, and d j For the second vector difference, s j For the first vector difference, α*d j +(1-α)*exp(s j ) represents the vector distance corresponding to the third vector. In the above formula (5), exp() is used to smooth the influence of the difference of the first vector and avoid extreme results caused by the difference between the first vector and the second vector being too large. It should be noted that in the above formula (5), the maximum value of the vector distance corresponding to the third vector is selected as the text difference degree between the first text and the second text. In actual implementation, the average value of the vector distance corresponding to the third vector can also be obtained by summing the vector distances corresponding to the third vector and then averaging them, and this average value can be used as the text difference degree between the first text and the second text. The specific method for determining the text difference degree between the first text and the second text can be selected according to the actual situation, and no specific limitation is made here.

[0099] The above method can improve the accuracy of the determined textual differences between the first and second texts.

[0100] In some embodiments, the determination of the first weight of the first vector difference and the second weight of the second vector difference can be achieved by the following technical solution: determining the number of first dimensions of the first vector and the number of second dimensions of the second vector; using the ratio of the number of second dimensions to the number of first dimensions as the first weight of the first vector difference; and determining the second weight of the second vector difference based on the first weight.

[0101] In practice, the method for determining the first weight can be found in the following formula (6):

[0102]

[0103] In formula (6), α is the first weight, n is the number of the first dimension, and m is the number of the second dimension. The larger the value of α, the more information each third vector of the first vector contains, and the trend of change should occupy a larger weight. Conversely, the smaller the value of α, the less information each third vector contains, and the degree of deviation should occupy a larger weight.

[0104] In practice, after obtaining the first weight, the result of 1 minus the first weight can be used as the second weight.

[0105] By setting a first weight, the robustness and flexibility of the text difference calculation method are improved through the above approach.

[0106] In practical implementation, after determining the text differences between multiple first and second texts, the texts can be clustered based on these differences. Specifically, each text can be treated as a separate cluster center, and texts with text differences exceeding a threshold can be continuously merged until a stopping condition is met, such as reaching a preset number of clusters or a minimum text difference threshold between clusters. Alternatively, all texts can be initially treated as a single cluster, and texts with lower text differences can be gradually separated from the cluster to form new clusters. Clustering can also be performed based on the density distribution of texts in the text difference space. If the text density in a certain region exceeds a set threshold, these texts are grouped into a single cluster. Low-density, isolated texts may be considered noise points. This algorithm can discover clusters of arbitrary shapes and does not require pre-specifying the number of clusters. Alternatively, the number of clusters K can be pre-specified, and the algorithm will randomly initialize K cluster centers, then assign each text to the cluster containing the cluster center with the highest text difference, and continuously update the cluster centers until the cluster centers no longer change significantly or the maximum number of iterations is reached.

[0107] In practical implementation, in customer service scenarios, after determining the text difference between the user's input inquiry (i.e., the first text) and the various questions stored in the database (i.e., the second text), an appropriate text difference threshold is set according to business needs. When the text difference between the user's inquiry and the questions in the database exceeds this threshold, a matching question is considered found. For example, if the text difference threshold is set to 0.8, a match is determined when the calculated text difference reaches or exceeds 0.8. Alternatively, the text difference between the user's inquiry and all questions in the database can be calculated, and then the questions in the database can be sorted from highest to lowest text difference. From the sorted question list, questions with a text difference exceeding the set threshold are selected as preliminary matching results. Among the preliminary matching results, the question with the highest text difference is selected as the best matching question. If multiple questions have the same and highest text difference, further filtering can be performed based on factors such as the frequency of occurrence and update time of the questions. Once the best matching question is determined, the answer corresponding to the question is extracted from the database and presented to the user in an appropriate way, such as displaying the answer directly on the online customer service interface or informing the user through voice broadcast.

[0108] Setting this threshold requires some experimentation and adjustment. If it is too high, the matching rate may be low, and many user questions will not be able to find corresponding answers; if it is too low, it may introduce a large number of inaccurate matching results.

[0109] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0110] Frequently Asked Questions (FAQ) systems are the most common method used in intelligent customer service. After a customer enters a query question into the intelligent dialogue system, the backend recognizes the semantic information of the text and returns the answer. The underlying algorithm logic typically involves first collecting business data and user log data to maintain an FAQ database, which stores multiple question-answer pairs. Then, standard questions in the question-answer pairs are expanded to generate multiple similar questions for semantic matching. When a customer enters a query question, the system first performs a coarse recall, searching the FAQ database for a list of similar questions with a high degree of matching. Then, a more precise large model is used for fine ranking; if the number of similar questions exceeds a set threshold, the relevant answer is returned to the customer.

[0111] Generally, in a question-answering database, each question is a short string (the second text mentioned above). For example, in e-commerce, common questions include "air conditioner won't start," "air conditioner isn't cooling," "shoes don't fit," and "clothes size is incorrect." When a user enters a short query, existing text matching methods can match it to a question-answering pair with high textual difference, thus returning an accurate answer. However, when the user enters a longer query (the first text mentioned above), these methods will give a lower textual difference, failing to return an accurate answer. For example, the query: "I bought a pair of shoes from you last week, and they arrived yesterday. I tried them on, and they feel a bit small. I always wear this size. Is there a problem with your sizing? How can I fix this?" The answer is: "Shoes don't fit." In this case, the query and the question have a high semantic textual difference. However, because the query is long and contains too much redundant information, existing methods consider the textual difference to be low, thus failing to match the correct question and return an accurate answer.

[0112] On this basis, in the data processing method provided in the embodiments of the present application, a vector representation model may be first trained for each of a question text library composed of questions and an answer text library composed of answers in a question-answer library; when a user inputs a query question, it is represented using the long text vector representation model; the answer is represented using the short text vector representation model; then, the text difference value between the two texts is calculated by using the method for calculating the text difference between text vectors proposed in the present solution.

[0113] Specifically, the text vector representation models are trained first, and a vector representation model is respectively trained for a short text (i.e., the second text mentioned above) and a long text (i.e., the first text mentioned above). It is assumed that the dimension of the short text vector representation (i.e., the second number of dimensions mentioned above) is n dimensions, and the dimension of the long text vector representation (i.e., the first number of dimensions mentioned above) is m dimensions, where n<m; a word vector model or other models may be adopted as the vector representation model.

[0114] A short text is input, and represented as an n-dimensional vector using the short text vector representation model, which is assumed to be A=(A1,A2,…,A n ); a long text is input, and represented as an m-dimensional vector using the long text vector representation model, which is assumed to be B=(B1,B2,…,B m ).

[0115] Thereafter, the text difference between the short text and the long text is determined through the following formula (7) to formula (10).

[0116] S(A,B)=max(α*d j +(1-α)*exp(s j )) (7)

[0117] In formula (7), S(A,B) is the text difference between the first text and the second text, α is a first weight, which can be specifically obtained through the following formula (8), d j is a second vector difference, which can be specifically obtained through the following formula (9), s j is a first vector difference, which can be specifically implemented through the following formula (10).

[0118]

[0119] In formula (8), α is the first weight, n is the first number of dimensions, and m is the second number of dimensions.

[0120]

[0121] In formula (9), d j is the second vector difference, A i+1Let A be the (i+1)th first element value in the second vector, where i is the number of first element values. i Let |A| be the value of the i-th first element in the second vector. i+1 -A i | represents the difference of the second element, max(A) i+1 A i ) is the first maximum value. As the first ratio, C j,i+1 The j-th third vector contains the (i+1)-th second element value, where i is the number of second element values, and C j,i Let |C be the value of the second element of the j-th third vector, |C j,i+1 -C j,i | represents the difference of the third element, max(C) j,i+1 C j,i ) is the second maximum value. This is the second ratio.

[0122]

[0123] In formula (), s j For the first vector difference, A i Let C be the value of the first element of the i-th vector. j,i Let σ(A) be the value of the i-th second element of the j-th third vector, and let σ(A) represent the standard deviation of the second vector; ∈ is a small positive number to avoid the denominator being 0.

[0124] The above methods can achieve the following beneficial effects:

[0125] First, for the scenario of "short text - long text" text matching, the problem of calculating the text difference between text vectors of different dimensions is solved by using different dimensions to represent short text and long text. By combining numerical difference (the degree of deviation in n-dimensional space) and trend difference (the average value of the trend of change in n-dimensional space) to determine the text difference between vectors, it can reflect the degree of matching of the numerical values ​​of two vectors and reflect the consistency of the trend of change, effectively solving the shortcomings of existing methods. At the same time, by setting adaptive weights when determining the text difference between texts, the robustness and flexibility of the determined text difference are improved.

[0126] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0127] The following description continues to illustrate the exemplary structure of the data processing apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software module in the data processing device 555 stored in the memory 550 may include:

[0128] The decomposition module 5551 is used to decompose the first vector of the first text to obtain multiple third vectors with the same number of dimensions as the second vector of the second text.

[0129] Calculation module 5552 is used to perform numerical difference calculation processing on the second vector and each of the third vectors to obtain a first vector difference between the second vector and each of the third vectors;

[0130] The determining module 5553 is used to determine the first element change data in the second vector based on the multiple first element values ​​included in the second vector, and to determine the second element change data in the third vector based on the multiple second element values ​​included in the third vector;

[0131] The determining module 5553 is further configured to determine the second vector difference between the second vector and each of the third vectors based on the first element change data and the second element change data corresponding to each of the third vectors;

[0132] The fusion module 5554 is used to perform fusion processing on the first vector difference and the second vector difference to obtain the text difference degree between the first text and the second text.

[0133] In some embodiments, the decomposition module 5551 is further configured to, if the text length of the first text is greater than the text length threshold, call a model corresponding to the first dimension number to perform vectorization processing on the first text to obtain a first vector with the first dimension number; if the text length of the second text is not greater than the text length threshold, call a model corresponding to the second dimension number to perform vectorization processing on the second text to obtain a second vector with the second dimension number.

[0134] In some embodiments, the decomposition module 5551 is further configured to subtract the number of the first dimension of the first vector from the number of the second dimension of the second vector, and add one to the subtraction result to obtain a first number; and to decompose the first vector to obtain the first number of third vectors having the second number of dimensions.

[0135] In some embodiments, the calculation module 5552 is further configured to perform the following processing for each of the third vectors: determine the first element difference between the nth first element value in the second vector and the nth second element value in the third vector, where n takes the value from 1 to N; perform fusion processing on the N first element differences to obtain a first fusion result; and determine the first vector difference between the second vector and the third vector based on the standard deviation of the second vector and the first fusion result.

[0136] In some embodiments, the determining module 5553 is further configured to determine the second element difference between the (m+1)th first element value and the mth first element value in the second vector, where m ranges from 1 to N-1; determine the first ratio between the second element difference and the first maximum value as the first element change data corresponding to the mth first element value, wherein the first maximum value is the larger value between the (m+1)th first element value and the mth first element value; and form the first element change data corresponding to the N-1 first element values ​​into the first element change data within the second vector.

[0137] In some embodiments, the determining module 5553 is further configured to perform the following processing for each of the third vectors: determining the third element difference between the (m+1)th second element value and the mth second element value in the third vector, where m takes the value from 1 to N-1; determining the second ratio between the third element difference and the second maximum value as the second element change data corresponding to the mth second element value, wherein the second maximum value is the larger value between the (m+1)th second element value and the mth second element value; and forming the second element change data within the third vector by combining the second element change data corresponding to the N-1 second element values.

[0138] In some embodiments, the determining module 5553 is further configured to perform the following processing for each of the third vectors: determine the second element difference between the first element change data corresponding to the m-th first element value of the second vector and the second element change data corresponding to the m-th second element value of the third vector, where m takes the value from 1 to N-1; perform fusion processing on N-1 second element differences to obtain the second vector difference between the second vector and the third vector.

[0139] In some embodiments, the fusion module 5554 is further configured to determine a first weight of the first vector difference and a second weight of the second vector difference; and perform the following processing for each third vector: based on the first weight and the second weight, perform fusion processing on the first vector difference between the second vector and the third vector and the second vector difference between the second vector and the third vector to obtain the vector distance between the second vector and the third vector; and take the maximum text difference degree among the vector distances corresponding to the plurality of third vectors as the text difference degree between the first text and the second text.

[0140] In some embodiments, the fusion module 5554 is further configured to determine the number of first dimensions of the first vector and the number of second dimensions of the second vector; use the ratio of the number of second dimensions to the number of first dimensions as a first weight of the difference between the first vectors; and determine a second weight of the difference between the second vectors based on the first weight.

[0141] This application provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application embodiment.

[0142] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will perform the data processing method provided in this application, for example... Figure 3 The data processing method is shown.

[0143] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0144] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0145] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0146] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0147] In summary, the following beneficial effects can be achieved through the embodiments of this application:

[0148] The first vector of the first text is decomposed to obtain multiple third vectors with the same number of dimensions as the second vector of the second text. By decomposing the vectors, multiple sets of vectors with the same dimensions are obtained. Since the vectors themselves are not modified, the information contained in the vectors is preserved, thus ensuring the accuracy of the text difference degree determined subsequently between the texts. The numerical difference between the second vector and each of the third vectors is calculated to obtain the first vector difference between the second vector and each of the third vectors. Based on the multiple first element values ​​included in the second vector, the change data of the first element within the second vector is determined, and based on the multiple second element values ​​included in the third vector, the change data of the second element within the third vector is determined. Based on the change data of the first element and the change data of the second element corresponding to each of the third vectors, the second vector difference between the second vector and each of the third vectors is determined. The first vector difference and the second vector difference are fused to obtain the text difference degree between the first text and the second text. By combining the text difference degree between vectors and the difference in the change of element values ​​within vectors, the text difference degree between the first text and the second text is determined, improving the accuracy of the determined text difference degree.

[0149] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: The first vector of the first text is decomposed to obtain multiple third vectors with the same number of dimensions as the second vector of the second text; The numerical difference between the second vector and each of the third vectors is calculated to obtain the first vector difference between the second vector and each of the third vectors. Based on the multiple first element values ​​included in the second vector, determine the first element change data within the second vector, and based on the multiple second element values ​​included in the third vector, determine the second element change data within the third vector; Based on the first element change data and the second element change data corresponding to each of the third vectors, the second vector difference between the second vector and each of the third vectors is determined; The first vector difference and the second vector difference are fused to obtain the text difference degree between the first text and the second text.

2. The method according to claim 1, characterized in that, Before decomposing the first vector of the first text to obtain multiple third vectors with the same dimensions as the second vector of the second text, the method further includes: If the length of the first text is greater than the text length threshold, the model corresponding to the first dimension number is called to vectorize the first text to obtain a first vector with the first dimension number. If the length of the second text is not greater than the text length threshold, the model corresponding to the number of second dimensions is called to vectorize the second text, resulting in a second vector with the number of second dimensions.

3. The method according to claim 1, characterized in that, The second vector includes N first element values, and the third vector includes N second element values, where N is a positive integer. The step of calculating the numerical difference between the second vector and each of the third vectors to obtain the first vector difference between the second vector and each of the third vectors includes: For each of the third vectors, the following processing is performed: Determine the difference between the first element value of the nth first element in the second vector and the second element value of the nth second element in the third vector, where n ranges from 1 to N; The differences among the N first elements are fused to obtain the first fusion result; Based on the standard deviation of the second vector and the first fusion result, a first vector difference between the second vector and the third vector is determined.

4. The method according to claim 1, characterized in that, The second vector includes N first element values, where N is a positive integer. The step of determining the first element change data within the second vector based on the multiple first element values ​​included in the second vector includes: Determine the difference between the (m+1)th first element value and the mth first element value in the second vector, where m ranges from 1 to N-1; The first ratio between the difference of the second element and the first maximum value is determined as the first element change data corresponding to the m-th first element value, wherein the first maximum value is the larger value between the (m+1)-th first element value and the m-th first element value; The first element change data corresponding to N-1 first element values ​​are combined to form the first element change data in the second vector.

5. The method according to claim 1, characterized in that, The third vector includes N second element values, where N is a positive integer. Determining the second element change data within the third vector based on the multiple second element values ​​included in the third vector includes: For each of the third vectors, the following processing is performed: Determine the difference between the (m+1)th second element value and the mth second element value in the third vector, where m ranges from 1 to N-1; The second ratio between the difference of the third element and the second maximum value is determined as the second element change data corresponding to the m-th second element value, wherein the second maximum value is the larger value between the (m+1)-th second element value and the m-th second element value; The second element change data corresponding to N-1 second element values ​​are combined to form the second element change data in the third vector.

6. The method according to claim 1, characterized in that, The second element change data within the third vector includes N-1 second element change data corresponding to the second element values, and the first element change data within the second vector includes N-1 first element change data corresponding to the first element values. The step of determining the second vector difference between the second vector and each of the third vectors based on the first element change data and the second element change data corresponding to each of the third vectors includes: For each of the third vectors, the following processing is performed: Determine the difference between the first element change data corresponding to the m-th first element value of the second vector and the second element change data corresponding to the m-th second element value of the third vector, where m ranges from 1 to N-1; The differences of the second element (N-1) are fused to obtain the second vector difference between the second vector and the third vector.

7. The method according to claim 1, characterized in that, The process of fusing the first vector difference and the second vector difference to obtain the text difference degree between the first text and the second text includes: Determine the first weight of the first vector difference and the second weight of the second vector difference; For each of the third vectors, the following processing is performed: Based on the first weight and the second weight, the first vector difference between the second vector and the third vector and the second vector difference between the second vector and the third vector are fused to obtain the vector distance between the second vector and the third vector; The maximum text difference among the vector distances corresponding to the multiple third vectors is taken as the text difference between the first text and the second text.

8. The method according to claim 7, characterized in that, Determining the first weight of the first vector difference and the second weight of the second vector difference includes: Determine the number of the first dimension of the first vector and the number of the second dimension of the second vector; The ratio of the number of the second dimension to the number of the first dimension is used as the first weight of the difference of the first vector; The second weight of the difference between the second vectors is determined based on the first weight.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the data processing method according to any one of claims 1 to 8.

10. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 8.

Images