Reply information generation method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present application provide a reply information generation method and device, electronic equipment, storage medium and program product. The method responds to the acquisition of the commodity corresponding consultation information, filters the initial picture set matched with the consultation information in the picture set corresponding to the commodity, realizes the semantic matching of the consultation information and each picture in the picture set, avoids the problem of inaccurate recognition result caused by text recognition on the picture; on this basis, a semantic correlation model is called to determine the target picture according to the semantic correlation between each initial picture and the consultation information, ensure the accuracy of the target picture and the consultation information in the semantic level, and generate and push the reply information containing the target picture under the condition that the target picture and the consultation information are matched, avoid the situation that the reply content and the consultation information are not matched.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, storage medium, and program product for generating response information. Background Technology

[0002] With the development of technology, online dialogue services have the ability to handle various consumer questions. For example, in response to consumer inquiries about product-related information, text recognition can be performed on product images, and the reply text can be determined based on the recognition results.

[0003] However, due to the high complexity of images, the accuracy of text recognition results for images is low, and the feedback content is only text, which is monotonous, resulting in poor accuracy and reliability. Summary of the Invention

[0004] To overcome the problems existing in related technologies, embodiments of this application provide a method, apparatus, electronic device, storage medium, and program product for generating response information.

[0005] According to a first aspect of the embodiments of this application, a method for generating response information is provided, the method comprising: In response to obtaining the consultation information corresponding to the product, an initial image set matching the consultation information is filtered from the image set corresponding to the product; The semantic relevance model is invoked to determine the target image based on the semantic relevance between each initial image and the consultation information; If the target image and the consultation information match, a response containing the target image is generated and pushed to the system.

[0006] According to a second aspect of the embodiments of this application, a response information generation apparatus is provided, the apparatus comprising: The filtering module is used to filter an initial set of images that match the consultation information in the image set corresponding to the product in response to obtaining consultation information. The relevance module is used to call the semantic relevance model to determine the target image based on the semantic relevance between each initial image and the consultation information; The verification module is used to generate and push a response containing the target image if the target image and the consultation information match.

[0007] According to a third aspect of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the computer program being executed by the processor to cause the electronic device to perform the method as described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the method as described in the first aspect.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0010] The technical solutions provided in this application embodiment may include the following beneficial effects: The response information generation method in this embodiment, in response to obtaining consultation information corresponding to a product, filters an initial image set that matches the consultation information from the image set corresponding to the product, thus achieving semantic matching between the consultation information and each image in the image set, avoiding the problem of inaccurate recognition results caused by text recognition of images; on this basis, a semantic relevance model is invoked to determine the target image based on the semantic relevance between each initial image and the consultation information, ensuring the accuracy of the target image and the consultation information at the semantic level, and generating and pushing response information containing the target image when verifying that the target image and the consultation information match, thus avoiding the situation where the response content does not match the consultation information.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below. It should be understood that those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0013] Figure 1 This is a schematic diagram of a scenario corresponding to the response information generation method provided in the embodiments of this application; Figure 2 An exemplary flowchart corresponding to the response information generation method provided in the embodiments of this application; Figure 3 An exemplary flowchart corresponding to another response information generation method provided in the embodiments of this application; Figure 4 An exemplary schematic diagram of the response information generation apparatus provided in the embodiments of this application; Figure 5 An exemplary schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.

[0015] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments and is not intended to limit the technical solutions of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise.

[0016] It should also be understood that although the terms "first," "second," etc., may be used in the following embodiments to describe a certain type of object, the objects should not be limited to these terms. These terms are used to distinguish the specific implementation objects of that type of object.

[0017] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] The technical scenarios of the embodiments of this application are described below.

[0019] With the development of technology, online dialogue services have the ability to handle various consumer questions. For example, in response to consumer inquiries about product-related information, text recognition can be performed on product images, and the reply text can be determined based on the recognition results.

[0020] However, due to the high complexity of images—such as the inclusion of marketing terms, decorative words, repetitive fields, and potential text formatting issues, misalignments, typos, and missing units—text recognition in images suffers from high noise levels, leading to inaccurate results. Furthermore, the input content is limited to text, resulting in poor accuracy and reliability.

[0021] In view of this, embodiments of this application propose a method for generating response information, which can be applied to a response information generation system. For example... Figure 1 As shown, this is a response information generation system that implements a response information generation method. Figure 1As shown, the response information generation system includes a client 101 and a server 102, which are connected via a network. The client 101 can be hardware, such as a mobile phone, personal computer, tablet computer, wearable device, or other electronic device, or it can be an application (APP) or software module deployed on an electronic device. The client 101 serves as an online service portal for users. Users can log in to this portal through electronic devices or applications and send inquiries to the server 102. The server 102 can be an application, service, instance, software module, virtual machine, container, or cloud server, or a hardware device or chip with data processing capabilities. After receiving the inquiry information, the server 102 processes it to generate the corresponding response information.

[0022] It should be understood that the above description of each functional layer of the response information generation system is merely illustrative and does not constitute a limitation on the response information generation system in this application embodiment. In actual implementation scenarios, the response information generation system may include more or fewer structures, and this application embodiment does not limit this.

[0023] It should be understood that Figure 1 The illustrated application scenario diagram is merely an illustrative representation of the response information generation system involved in the embodiments of this application and does not constitute a limitation on the technical solutions of the embodiments of this application.

[0024] In the above operating environment, the embodiments of this application provide as follows: Figure 2 The response information generation method shown is applicable to, for example, the response information generation method of ... Figure 1 The corresponding server is 102. The method for generating this response information includes the following steps: In step S201, in response to obtaining the consultation information corresponding to the product, an initial image set matching the consultation information is filtered from the image set corresponding to the product.

[0025] For example, the consultation information corresponding to a product typically refers to the questions or needs described by users regarding a specific product in e-commerce customer service conversations. This can be a short question or a phrase, such as "How much memory does this phone have?", "Material composition", or "Installation steps". Essentially, it's a textual expression of the user's intent. The image set corresponding to a product refers to the collection of all searchable image materials associated with that product. In e-commerce scenarios, this usually refers to the product details page images, the content of which may include, but is not limited to: product appearance images, parameter specification tables, size charts, material composition labels, functional diagrams, installation step diagrams, packaging information images, product comparison images, etc. The initial image set refers to a smaller subset of images selected from the product image set after preliminary semantic matching, ranked higher in relevance to the consultation information. For example, it could be the top 5 or top 10 images with the highest similarity. Its purpose is to reduce the computational scope of subsequent, more refined, and time-consuming processing steps (such as reordering and consistency checks), thereby improving efficiency.

[0026] After receiving a user's inquiry related to a certain product, a preliminary subset of images that semantically matches the inquiry is selected from a pre-determined set of images associated with the product. This subset is the initial image set.

[0027] The similarity metric used to filter the initial image set can be cosine similarity, Euclidean distance, inner product, etc., and the sorting strategy can be to select all images with a similarity greater than a certain threshold, or to return a fixed number of images in a preset sorting order. Those skilled in the art can choose one or more of the above filtering methods in combination according to the specific scenario, performance requirements, and accuracy targets, and this invention does not limit this.

[0028] By conducting preliminary screening, subsequent operations are limited to a smaller candidate set, which reduces response latency and computational overhead to some extent.

[0029] In step S202, the semantic relevance model is invoked to determine the target image based on the semantic relevance between each initial image and the consultation information.

[0030] For example, a semantic relevance model refers to a model trained specifically to determine whether a piece of text and an image semantically match, and the degree of matching. In this embodiment, it can refer to a generative visual language model whose input is "user inquiry information" and "each initial image," and whose output can be a direct judgment of relevance (such as "yes / no"), a continuous score representing relevance, or the ranking position among the initial images. Semantic relevance refers to a deeper level of association, requiring not only that the image contains the keywords or objects mentioned in the inquiry information, but also that the image content can directly and effectively answer or support the questions implied in the inquiry information. For example, for the inquiry "fill weight," a table image containing the explicit field "fill weight: 200g" has high semantic relevance; while an image that only shows the appearance of a down jacket but does not label the parameters has low semantic relevance. The target image refers to one (or several) images that, after being rearranged and judged by the semantic relevance model, are confirmed to be suitable for display to the user.

[0031] After obtaining the initial image set through preliminary screening, a semantic relevance discrimination model is invoked to perform semantic relevance analysis and scoring on each image in the initial image set, in conjunction with the user's consultation information. Finally, based on the relevance judgment results output by this model, one or more images are selected from the initial image set as the final "target images" to be output. By using a semantic relevance discrimination model with stronger discriminative capabilities, the initial image set after coarse screening is refined and sorted to select images that match the consultation information, thus solving the problem that relying solely on vector similarity thresholds is insufficient to balance accuracy and coverage.

[0032] In step S203, if the target image and the consultation information match, a response message containing the target image is generated and pushed.

[0033] For example, before using the target image as the response information, it is necessary to verify again whether the target image matches the consultation information to avoid the response information including a target image that does not contain relevant information corresponding to the consultation intent in the consultation information. If it is determined that the target image matches the consultation information, a response information containing the target image is generated, and the target image is used as supporting image in the response information.

[0034] Of course, if the target image is not matched in any of the above steps, or if the target image does not match the consultation information, a response message without the target image can be generated.

[0035] As can be seen, the response information generation method disclosed in this application, in response to obtaining the consultation information corresponding to the product, filters the initial image set that matches the consultation information from the image set corresponding to the product, realizing semantic matching between the consultation information and each image in the image set, avoiding the problem of inaccurate recognition results caused by text recognition of images; on this basis, a semantic relevance model is called to determine the target image based on the semantic relevance between each initial image and the consultation information, ensuring the accuracy of the target image and the consultation information at the semantic level, and generating and pushing response information containing the target image when verifying that the target image and the consultation information match, avoiding the situation where the response content and the consultation information do not match.

[0036] In the process of selecting the initial image set, if the selection is based solely on the similarity threshold between each image set and the consultation information, the setting of the similarity threshold becomes crucial. A threshold that is too high may miss correct target images, while a threshold that is too low will retain a large number of unrelated images, resulting in numerous noisy images during the subsequent determination of target images. Therefore, in some embodiments, the process of determining the initial image set can also be achieved by: extracting the semantic feature vector of the consultation information and extracting the image feature vectors of each image in the corresponding product image set, where the semantic feature vector and image feature vector are feature vectors mapped to the same dimension; calculating the similarity between the semantic feature vector and the image feature vector of each image; and determining a predetermined number of images corresponding to the image feature vectors as the initial image set, in descending order of similarity.

[0037] For example, semantic feature vectors are high-dimensional vector representations obtained by encoding user-input consultation information using a multimodal model, abstracting the semantic information of the text. Image feature vectors are high-dimensional vector representations obtained by encoding each image in an image set using the same multimodal model (or its image encoding branch), abstracting the visual semantic information of the images. Since semantic feature vectors and image feature vectors have the same length (number of dimensions), they can be mathematically processed using operations such as dot product and cosine similarity.

[0038] Semantic feature vectors of the consultation information and image feature vectors of each image in the image set are extracted separately, ensuring that these two types of vectors lie in the same feature space. Next, the similarity between the semantic vector of the consultation information and the image vectors of all images is calculated. Finally, the images are sorted according to the calculated similarity values, and a predetermined number of images with the highest similarity are selected to form the initial image set.

[0039] Pre-trained models such as Contrastive Language–Image Pre-training (CLIP) and General Multimodal Embedding (GME) can be used to pre-generate image feature vectors for each image in the image set and establish a vector index library. This allows for the extraction of semantic feature vectors corresponding to the received consultation information after receiving the consultation information. Once the semantic feature vectors are extracted, the similarity can be calculated directly by reading the image feature vectors of each image in the image set, thus improving response speed.

[0040] To ensure that semantic feature vectors and image feature vectors map to feature vectors of the same dimension, the same multimodal coding model can be used to generate the corresponding feature vectors. Alternatively, the text encoder and image encoder can be trained separately to ensure that the output vectors of the trained text encoder and image encoder have the same dimension, thereby achieving spatial feature alignment. This application does not limit the implementation method of the above feature alignment; those skilled in the art can set it according to the actual situation.

[0041] By mapping text and images to the same vector space and utilizing vector operations, millisecond-level initial screening was achieved, improving response speed to some extent. The multimodal semantic matching method effectively understands the text and image content, recalls images highly relevant to the consultation intent, and provides high-quality candidates for subsequent processes.

[0042] Building upon the above approach of selecting a preset number of images as the initial image set based on similarity magnitude to address the challenge of setting a similarity threshold, to more accurately ensure a strong correlation between the target image and the consultation information, in some embodiments, and to reduce the computational burden of correlation calculations, for any initial image in the initial image set, if the similarity of the initial image is greater than a preset similarity threshold, then the semantic correlation between the initial image and the consultation information is calculated; the initial image whose semantic correlation satisfies the preset correlation condition is then selected as the target image.

[0043] For example, a preset similarity threshold refers to a filtering threshold set for similarity. Its function is to quickly filter out images that have a very low semantic relevance to the consultation information, avoiding sending a large number of obviously irrelevant images into the subsequent, more computationally expensive re-ranking model, thereby improving overall response efficiency. A preset relevance condition refers to a passing condition set for semantic relevance. This can be a discrimination threshold (e.g., the probability of the model outputting "YES" must be greater than 0.5), or it can be simply "the model's discrimination result is YES," or it can be an initial image with a preset relevance score, determining which images are ultimately eligible for output.

[0044] For each image in the initial image set obtained from the preliminary screening, it is first determined whether its similarity to the consultation information is greater than a preset similarity threshold. If the condition is met, the image and consultation information are input into a semantic relevance model (such as a rearrangement model) to calculate its detailed relevance score or discrimination result for "whether it can be used as an image". Ultimately, only those initial images whose semantic relevance discrimination results meet the preset conditions (e.g., a discrimination of "YES" or a score higher than a certain threshold) will be identified as the final target images.

[0045] The specific values ​​of the preset similarity threshold and preset relevance condition (threshold) can be dynamically adjusted and A / B tested according to different requirements for coverage and accuracy in the task scenario. The semantic relevance model can be fine-tuned using different large visual language models as a base. Those skilled in the art can flexibly choose Pairwise, Listwise, or other discrimination methods, or use them in combination, according to online latency, computing resources, and requirements for decision interpretability; this invention does not limit this.

[0046] In the process of verifying the target image, in addition to verifying that the target image corresponds to the consultation intent in the consultation information, it is also necessary to ensure that the target image is consistent with other information in the response information. The other information can be the response text. Therefore, in some embodiments, the matching of the target image and the consultation information can be determined by the following steps: if the target image includes a text block corresponding to the consultation intent in the consultation information, and the semantics of the text block in the target image are the same as the semantics of the pre-obtained response text, then the target image and the consultation information are determined to match; the response text refers to the text response content corresponding to the consultation information.

[0047] For example, the text block corresponding to the consultation intent in the consultation information refers to the text area in the target image that is directly related to the core intent of the consultation information. For example, if the consultation information is "how many grams of down filling," then the table cell or text area in the target image labeled "down filling: 200g" is this "text block." This text block is not equivalent to all the text in the entire image (OCR result), but specifically refers to the part of the visual text that carries the key fields of the answer. The pre-acquired response text refers to the plain text content generated and determined before the image-text matching process through large language models or knowledge base retrieval, used to directly answer the consultation information. This response text is the known input. Semantic similarity means that they are equivalent in meaning and do not conflict. For example, if the response text is "This underwear contains 55% other materials," and the image text block is "Other materials: 55%," although the wording is slightly different, the information conveyed about "composition and its proportion" is consistent, and they can be considered semantically similar.

[0048] When determining whether a target image matches the consultation information, two matching checks are performed. First, it checks whether the image contains text blocks corresponding to the consultation intent. Second, assuming the first check is satisfied, it further determines whether the semantics of the relevant text blocks in the image are semantically identical to the pre-acquired text response generated for the consultation information. Only when both conditions are met is the target image ultimately determined to match the consultation information.

[0049] Through the above two-stage matching, we ensure that the content of the target image strictly serves the answer, avoiding situations where the image does not match the text or fails to convey the meaning.

[0050] The semantic relevance model in the above embodiments is trained in the following way: constructing a set of text samples and an image sample set; the image sample set includes positive image samples and negative image samples; training the semantic relevance model to be trained based on the text samples and the image sample set to obtain the semantic relevance model.

[0051] For example, a text sample refers to a text data unit used for model training. In e-commerce customer service scenarios, this is typically the user's query. An image sample set refers to a collection of image data used for model training. Positive image samples are images that are semantically highly relevant to a specific text sample and can effectively answer or support the text question. For example, if the text sample is "What is the size chart for this garment?", an image clearly showing the size chart for that garment would be a positive sample. Negative image samples are images that are semantically irrelevant to a specific text sample, cannot answer the question, or even contradict it. For example, for the aforementioned text sample, an image showing a model wearing the garment but without size information, or a size chart for a completely different style of garment, could both be considered negative samples.

[0052] In training the semantic relevance model, a text sample set and an image sample set are first constructed. The image sample set is further divided into positive and negative image samples. Next, the constructed text and image sample sets are used as training data. Finally, these training data are used to train the semantic relevance model, and by optimizing the model parameters, a mature model capable of determining the semantic relevance between text and images is obtained.

[0053] During the sample set construction process, manual annotators can determine the relevance of candidate images based on text samples (such as user questions) to accurately label positive and negative samples. This method offers the advantages of high-quality labels and low noise, providing a solid foundation for model performance. Alternatively, existing product information (such as titles and attributes) can be used to perform keyword matching with image text recognition results; images with high matching scores are selected as positive candidate images, and those with low matching scores are selected as negative candidate images. This application does not limit the method of constructing the sample set described above; those skilled in the art can set it according to actual circumstances. The specific sources of text and image samples can be historical customer service dialogue records from e-commerce platforms, product detail page data, etc. The ratio of positive to negative samples and the selection strategy for difficult negative samples can be flexibly configured and selected by those skilled in the art based on actual data conditions and performance requirements; this invention does not limit these aspects.

[0054] The training process can utilize pre-constructed image-text sample pairs to directly fine-tune a pre-trained visual language model end-to-end. The model input consists of text and images, and the output is a relevance score or classification label. Model parameters are updated by optimizing the loss function (such as cross-entropy loss or contrastive loss) through backpropagation. To further ensure the model's training effectiveness, human feedback or simulated user behavior can be introduced, employing preference optimization or reinforcement learning methods to train the model, making its output more consistent with expected relevance judgments.

[0055] In some embodiments, constructing a set of text samples and an image sample set includes: calculating the similarity between the text samples and each image sample in the image sample set; using image samples with a similarity greater than a preset similarity threshold as initial image samples; and dividing each initial image sample into positive image samples and negative image samples based on the semantics of the text patches included in each initial image sample.

[0056] For example, in constructing the sample set, for any text sample, it is necessary to construct corresponding positive and negative image samples. For each text sample, the similarity between the text sample and every image in the image sample library can be calculated. Then, images with similarity exceeding a preset threshold are selected to form an "initial image sample set." Finally, for each image in this initial set, the semantics of the text patches contained within it are analyzed to determine whether the image is a "positive" (relevant) sample or a "negative" (irrelevant) sample for answering the text question, thus completing the fine-grained classification of the samples.

[0057] In some embodiments, based on the semantics of the text tiles included in each initial image sample, each initial image sample is divided into positive image samples and negative image samples, including: for any initial image sample, if the initial image sample includes a text tile corresponding to the Chinese text intent in the text sample, then the initial image sample is a positive image sample; if the fields of the initial image sample do not include text tiles corresponding to the Chinese text intent in the text sample, then the initial image sample is a negative image sample; negative image samples with a similarity greater than a preset similarity threshold are determined as a first negative image sample set; negative image samples with a similarity less than a preset similarity threshold are determined as a second negative image sample set.

[0058] For example, the first negative image sample set refers to the set of negative samples that still have a high similarity (similarity greater than a threshold) to the text samples in terms of macro-semantic meaning. These samples are typically images that are "relevant to but do not answer" the question, such as images showing the same product but without key parameters; they are "hard negative samples" that the model easily confuses. The second negative image sample set refers to the set of negative samples that have a low similarity (similarity less than or equal to a threshold) to the text samples in terms of macro-semantic meaning. These samples are typically images that are clearly unrelated and belong to the category of "simple negative samples."

[0059] For any initially selected image sample, its classification as a positive or negative sample is based on whether it contains a specific text tile corresponding to the core intent of the text sample. Specifically, if the image contains a text tile that directly answers the question posed by the text sample, it is marked as a positive sample; otherwise, it is marked as a negative sample. Then, within the set of all negative samples, they are further divided into two subsets based on their similarity to the text sample and a preset threshold: those with similarity greater than the threshold are called the "first negative image sample set," and those with similarity less than or equal to the threshold are called the "second negative image sample set." In this embodiment, a first negative image sample set and a second negative image sample set are divided by a preset similarity threshold. During the training of the semantic relevance model, the semantic relevance model can be finely trained by the first negative image sample set of "difficult negative samples" to improve the accuracy of the semantic relevance model in recognizing "difficult negative samples".

[0060] In some embodiments, training a semantic relevance model to be trained based on text samples and image sample sets includes: training the semantic relevance model to be trained according to text samples, a first number of positive image samples in a positive image sample set, and a second negative image sample set to obtain a semantic relevance model to be enhanced; and enhancing the semantic relevance model to be enhanced using a second number of positive image samples in the text samples and positive image samples, and a first negative image sample set to obtain a semantic relevance model, wherein the second number is less than the first number and is a preset proportion of the number of the first negative image sample set.

[0061] For example, augmentation training refers to the second phase of training, starting with the model trained in the first phase and continuing training using more difficult negative samples. Its goal is to optimize the model parameters to handle more complex discrimination scenarios. The first quantity is the number of positive image samples used in the first phase of training. This quantity is typically large to ensure the model learns robust, query-relevant features from a sufficient number of positive examples in the early stages of training. The second quantity is the number of positive image samples used in the second phase of augmentation training. This quantity is smaller than the first quantity and is usually proportional to the number of difficult negative samples. This design aims to focus the model's attention more on learning how to distinguish difficult negative samples during the augmentation training phase, avoiding being "overwhelmed" by a large number of already learned simple positive samples.

[0062] The training of the semantic relevance model is divided into two stages. In the first stage, an initial model is trained using text samples, a "first number" of positive image samples selected from the positive image sample set, and all (or part) of the "second negative image sample set" (simple negative samples), resulting in a preliminarily optimized "semantic relevance model to be enhanced." In the second stage, the "semantic relevance model to be enhanced" obtained in the first stage is further enhanced using text samples, a "second number" of positive image samples selected from the positive image sample set, and all (or part) of the "first negative image sample set" (hard negative samples), ultimately yielding a more powerful "semantic relevance model." The "second number" of positive samples used in the second stage is less than the "first number" used in the first stage, and it represents a predetermined proportion of the number of samples in the "first negative image sample set."

[0063] This training process uses simple negative samples in the early stages to achieve stable convergence of the model, helping it master basic text-image recognition capabilities. In the later stages of training, the number of positive samples is reduced while the number of difficult negative samples is increased to improve the probability of the model accurately identifying positive samples among a large number of difficult negative samples, thereby optimizing the model.

[0064] For example, in the second stage of training, an image training set can be constructed using one positive image sample and four first negative image samples. The corresponding text samples and the image training set are then input into the semantic relevance model to be enhanced. The relevance score of each image sample in the image training set as a positive image sample is obtained, and the image samples that are identified as positive samples are then determined based on their relevance scores. If an image sample identified as a positive sample is indeed labeled as a positive image sample, the recognition is accurate; if an image sample identified as a positive sample is labeled as a negative image sample, the recognition is inaccurate.

[0065] In some embodiments, the above model training process can be implemented in the following way, wherein the training process for the first stage training and the second stage training is the same, the difference being that the input training samples are different.

[0066] In this embodiment of the disclosure, the target model is a semantic relevance model to be enhanced or a training model of a semantic relevance model. The training process of any target model includes: inputting a target image sample set and text samples corresponding to the target model into the target model to obtain a first relevance score sequence and a second relevance score sequence. Any first relevance score represents the relevance between a text sample and a corresponding positive image sample, and any second relevance score represents the relevance between a text sample and a corresponding negative image sample in the target image sample set; for any first relevance score, calculating the probability that the first relevance score is selected in the first relevance score sequence and the second relevance score sequence; using the probability set corresponding to the first relevance score sequence as the parameter to be optimized; and determining that the target model training is complete when the parameter to be optimized reaches a preset upper limit range.

[0067] For example, the target image sample set refers to a batch of image samples selected for the current evaluation or training iteration, including positive and negative image samples. The first / second relevance score sequence refers to the ordered list formed by the target model after scoring the positive and negative image samples respectively. The higher the relevance score in this list, the more relevant the model considers the image to be to the text sample. The probability of being selected refers to the probability that a positive sample (represented by its first relevance score) ranks better than all negative samples (represented by their second relevance scores) when ranked together. This is a metric measuring the "superiority" or "confidence" of the positive sample relative to all negative samples. The parameters to be optimized are not weights or biases within the model, but rather a set of external metrics used to monitor the training process and determine when to stop, namely the ranking probabilities of all positive samples. The preset upper limit range refers to a pre-set threshold or range. For example, it could be the average ranking probability of all positive samples being greater than 0.95, or all probability values ​​being greater than 0.9. Training is triggered to stop when the parameters to be optimized meet this condition.

[0068] During model training, a set of target image samples (typically containing positive and negative samples) and a text sample are input into the target model being trained. The model outputs a relevance score for each image sample, forming two sequences: a first relevance score sequence for positive image samples and a second relevance score sequence for negative image samples. Then, for each score in the first sequence, the probability that it will be selected as the highest score (or ranked highest) in the entire joint sequence (first sequence + second sequence) is calculated. This set of probabilities corresponding to all positive sample scores is used as a parameter to optimize the model's training performance. When these probability values ​​reach a preset upper limit (e.g., sufficiently close to 1), the model is considered sufficiently trained, and training can stop.

[0069] The process of generating the above response information will be summarized below with a specific example. Figure 3 As shown, after obtaining the consultation information, the semantic feature vector of the consultation information is extracted. If the image feature vectors of each image in the corresponding image set of the product are stored in the preset database, they can be read directly. If the image feature vectors of each image in the corresponding image set of the product are not stored in the preset database, the image feature vectors of each image in the image set are extracted. After obtaining the image feature vectors, the initial image set is selected based on the similarity between the image feature vectors and semantic feature vectors of each image. Further, the initial images with a preset similarity ranking in the initial image set are determined as target images. Finally, before generating the response information, the target image is double-verified: first, it is verified whether the target image contains the text block corresponding to the consultation intent in the consultation information; second, it is verified that the semantics of the text block in the target image are the same as the semantics of the pre-obtained response text. Under the premise that both verifications pass, the response information containing the target image is generated and pushed.

[0070] The above implementation steps can also be implemented through software modules to achieve the corresponding functions. Corresponding to the above response information generation method, this application embodiment can also provide a response information generation device.

[0071] like Figure 4 As shown, a response information generation device is provided, which is applied to a target device. The device may include: a filtering module 41, a relevance module 42, and a verification module 43.

[0072] The filtering module 41 is used to filter an initial set of images that match the consultation information in the image set corresponding to the product in response to obtaining the consultation information. The relevance module 42 is used to call the semantic relevance model to determine the target image based on the semantic relevance between each initial image and the consultation information; The verification module 43 is used to generate and push a response message containing the target image if the target image and the consultation information match.

[0073] In some optional embodiments, the filtering module 41 is further configured to: extract the semantic feature vector of the consultation information, and extract the image feature vector of each image in the image set corresponding to the product, wherein the semantic feature vector and the image feature vector are feature vectors mapped to the same dimension; Calculate the similarity between the semantic feature vector and the image feature vector of each image; Based on the order of similarity from highest to lowest, a predetermined number of images corresponding to the image feature vectors are selected as the initial image set.

[0074] In some optional embodiments, the relevance module 42 is further configured to: for any initial image, if the similarity of the initial image is greater than a preset similarity threshold, calculate the semantic relevance between the initial image and the consultation information; Initial images whose semantic relevance meets the preset relevance conditions are used as target images.

[0075] In some optional embodiments, the matching of the target image and the consultation information is determined through the following steps. If the target image includes a text block corresponding to the consultation intent in the consultation information, and the semantics of the text block in the target image are the same as the semantics of the pre-acquired response text, then the target image and the consultation information are determined to be a match; the response text refers to the text response content corresponding to the consultation information.

[0076] In some optional embodiments, the semantic relevance model is trained as follows: Construct text sample and image sample sets; the image sample set includes positive image samples and negative image samples; The semantic relevance model is trained based on the text samples and the image sample set to obtain the semantic relevance model.

[0077] In some optional embodiments, constructing the text sample and image sample sets includes: Calculate the similarity between each text sample and each image sample in the image sample set; Image samples with a similarity greater than a preset similarity threshold are used as initial image samples; Based on the semantics of the text blocks included in each of the initial image samples, each of the initial image samples is divided into positive image samples and negative image samples.

[0078] In some optional embodiments, dividing each initial image sample into positive image samples and negative image samples based on the semantics of the text tiles included in each initial image sample includes: For any initial image sample, if the initial image sample includes a text tile corresponding to the Chinese text intent in the text sample, then the initial image sample is a positive image sample; If the fields of the initial image sample do not include the text tile corresponding to the Chinese text intent in the text sample, then the initial image sample is a negative image sample; The negative image samples with a similarity greater than a preset similarity threshold are determined as the first negative image sample set; the negative image samples with a similarity less than the preset similarity threshold are determined as the second negative image sample set.

[0079] In some optional embodiments, training the semantic relevance model based on the text samples and the image sample set includes: The semantic relevance model to be trained is trained based on the text samples, the first number of positive image samples in the positive image sample set, and the second negative image sample set to obtain the semantic relevance model to be enhanced. The semantic relevance model to be enhanced is trained using text samples, a second number of positive image samples from the positive image samples, and the first set of negative image samples to obtain the semantic relevance model. The second number is less than the first number and is a preset proportion of the number of the first set of negative image samples.

[0080] In some optional embodiments, the target model is the semantic relevance model to be enhanced or the training model of the semantic relevance model, and the training process of any of the target models includes: The target image sample set corresponding to the target model and the text sample are input into the target model to obtain a first correlation score sequence and a second correlation score sequence. Any first correlation score represents the correlation between the text sample and the corresponding positive image sample, and any second correlation score represents the correlation between the text sample and the corresponding negative image sample in the target image sample set. For any first relevance score, calculate the probability that the first relevance score is selected in the first relevance score sequence and the second relevance score sequence; Use the probability set corresponding to the first correlation score sequence as the parameter to be optimized; When the parameters to be optimized reach the preset upper limit range, the training of the target model is determined to be complete.

[0081] As can be seen, by executing the response information generation device of this application embodiment, in response to obtaining the consultation information corresponding to the product, an initial image set matching the consultation information is filtered from the image set corresponding to the product, realizing semantic matching between the consultation information and each image in the image set, avoiding the problem of inaccurate recognition results caused by text recognition of images; on this basis, a semantic relevance model is invoked to determine the target image based on the semantic relevance between each initial image and the consultation information, ensuring the accuracy of the target image and the consultation information at the semantic level, and generating and pushing response information containing the target image when verifying that the target image and the consultation information match, avoiding the situation where the response content and the consultation information do not match.

[0082] Understandable, Figure 4 The division of the various modules is merely a logical functional division. In actual implementation, the functions of these modules can be integrated into the hardware entity of the electronic device.

[0083] Please refer to Figure 5 , Figure 5 An electronic device is provided, and this disclosure also provides an electronic device for performing some or all of the steps of the above-described response information generation method. Please refer to... Figure 5 The electronic device includes a processor 500, a memory 501, a bus 502, and a communication interface 503, wherein the processor 500, the communication interface 503, and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can run on the processor 500, and the processor 500 can be equivalent to a CPU, executing the aforementioned provisions of this disclosure when running the computer program. Figure 2 The response information generation method provided in any of the embodiments includes, during the execution of the aforementioned method, data and instructions are exchanged with the corresponding client through the communication interface 503.

[0084] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0085] Bus 502 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 501 is used to store programs, and the processor 500 executes the programs after receiving execution instructions. Figure 2 The illustrated embodiment reveals that the response information generation method can be applied to the processor 500, or implemented by the processor 500.

[0086] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a CPU, a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0087] The electronic device provided in this disclosure and the response information generation method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0088] This application also provides a computer-readable storage medium storing instructions for generating response information, which, when run on a computer, cause the computer to perform some or all of the steps in the method described in the foregoing embodiments.

[0089] This application also provides a computer program product including instructions for generating response information, which, when run on a computer, causes the computer to perform some or all of the steps in the method described in the foregoing embodiments.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0092] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, smartphone, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0096] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. A method for generating response information, characterized in that, The method includes: In response to obtaining the consultation information corresponding to the product, an initial image set matching the consultation information is filtered from the image set corresponding to the product; The semantic relevance model is invoked to determine the target image based on the semantic relevance between each initial image and the consultation information; If the target image and the consultation information match, a response containing the target image is generated and pushed to the system.

2. The method according to claim 1, characterized in that, The step of filtering the initial image set that matches the consultation information from the image set corresponding to the product includes: Extract the semantic feature vector of the consultation information, and extract the image feature vector of each image in the image set corresponding to the product. The semantic feature vector and the image feature vector are feature vectors mapped to the same dimension. Calculate the similarity between the semantic feature vector and the image feature vector of each image; Based on the order of similarity from highest to lowest, a predetermined number of images corresponding to the image feature vectors are selected as the initial image set.

3. The method according to claim 2, characterized in that, The step of determining the target image based on the semantic relevance between each initial image and the consultation information includes: For any initial image, if the similarity of the initial image is greater than a preset similarity threshold, then the semantic relevance between the initial image and the consultation information is calculated. Initial images whose semantic relevance meets the preset relevance conditions are used as target images.

4. The method according to claim 1, characterized in that, The following steps are used to determine if the target image and the consultation information match. If the target image includes a text block corresponding to the consultation intent in the consultation information, and the semantics of the text block in the target image are the same as the semantics of the pre-acquired response text, then the target image and the consultation information are determined to be a match; the response text refers to the text response content corresponding to the consultation information.

5. The method according to any one of claims 1-4, characterized in that, The semantic relevance model is trained in the following manner: Construct text sample and image sample sets; the image sample set includes positive image samples and negative image samples; The semantic relevance model is trained based on the text samples and the image sample set to obtain the semantic relevance model.

6. The method according to claim 5, characterized in that, The construction of the text sample and image sample sets includes: Calculate the similarity between each text sample and each image sample in the image sample set; Image samples with a similarity greater than a preset similarity threshold are used as initial image samples; Based on the semantics of the text blocks included in each of the initial image samples, each of the initial image samples is divided into positive image samples and negative image samples.

7. The method according to claim 6, characterized in that, The process of dividing each initial image sample into positive and negative image samples based on the semantics of the text patches included in each initial image sample includes: For any initial image sample, if the initial image sample includes a text tile corresponding to the Chinese text intent in the text sample, then the initial image sample is a positive image sample; If the fields of the initial image sample do not include the text tile corresponding to the Chinese text intent in the text sample, then the initial image sample is a negative image sample; The negative image samples with a similarity greater than a preset similarity threshold are determined as the first negative image sample set; the negative image samples with a similarity less than the preset similarity threshold are determined as the second negative image sample set.

8. The method according to claim 7, characterized in that, The step of training a semantic relevance model based on the text samples and the image sample set includes: The semantic relevance model to be trained is trained based on the text samples, the first number of positive image samples in the positive image sample set, and the second negative image sample set to obtain the semantic relevance model to be enhanced. The semantic relevance model to be enhanced is trained using text samples, a second number of positive image samples from the positive image samples, and the first set of negative image samples to obtain the semantic relevance model. The second number is less than the first number and is a preset proportion of the number of the first set of negative image samples.

9. The method according to claim 8, characterized in that, The target model is either the semantic relevance model to be enhanced or the training model of the semantic relevance model, and the training process of either target model includes: The target image sample set corresponding to the target model and the text sample are input into the target model to obtain a first correlation score sequence and a second correlation score sequence. Any first correlation score represents the correlation between the text sample and the corresponding positive image sample, and any second correlation score represents the correlation between the text sample and the corresponding negative image sample in the target image sample set. For any first relevance score, calculate the probability that the first relevance score is selected in the first relevance score sequence and the second relevance score sequence; Use the probability set corresponding to the first correlation score sequence as the parameter to be optimized; When the parameters to be optimized reach the preset upper limit range, the training of the target model is determined to be complete.

10. A response information generation device, characterized in that, The device includes: The filtering module is used to filter an initial set of images that match the consultation information in the image set corresponding to the product in response to obtaining consultation information. The relevance module is used to call the semantic relevance model to determine the target image based on the semantic relevance between each initial image and the consultation information; The verification module is used to generate and push a response containing the target image if the target image and the consultation information match.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

Images