Object processing method, device, storage medium and computer equipment
Through the multimodal model, multimodal information of objects is processed, combined with feature extraction, fusion and matching modules, the problems of small coverage and low recognition accuracy when searching for the same model are solved, and more efficient identification of the same model is achieved.
Patent Information
- Application Number
- CN202011624291.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-12-30
AI Technical Summary
In the prior art, the coverage range and recognition accuracy are small when searching for the same object, which fails to effectively solve this problem.
Using a multimodal model, by receiving multimodal information of objects (such as images, text, video, etc.), using feature extraction, fusion and matching modules, combined with modal sub-models, determine the same object, expand the coverage range of the same model and Improve recognition accuracy.
It realizes the expansion of coverage and improves recognition accuracy when searching for the same model, and solves the problems of small coverage and low recognition accuracy in the prior art.
Smart Images

Figure CN114764733B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and in particular to an object processing method, device, storage medium and computer equipment. Background Art
[0002] In the related art, when searching for the same object as an object (e.g., a product), generally some information (picture or text) of the object is input, and the search engine performs search processing based on the input information and outputs the same object as the object. However, when the above search method is used to search for the same object, the accuracy of the same object in the search results is poor, and the coverage of the same object is also small.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present invention provide an object processing method, apparatus, storage medium and computer device to at least solve the technical problems in the related art of small coverage of searching for the same item and low accuracy of identifying the same item.
[0005] According to a first aspect of an embodiment of the present invention, there is provided an object processing method, comprising: receiving description information of an object, wherein the description information includes multimodal information of the object; inputting the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a first data set, and the data in the first data set includes: the description information of the object and objects identical to the object.
[0006] Optionally, the multimodal information of the object includes at least one of the following: image, text, video, voice, and animation.
[0007] Optionally, the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same object as the object, including: extracting the features of the object in each modality through the feature extraction module; fusing the features in each modality through the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object as the object.
[0008] Optionally, the multimodal model also includes: a modal sub-model, which, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, also includes: inputting the modal data of the object in the modality into the modal sub-model to obtain the candidate same-type object of the object, wherein the modal sub-model is obtained through machine learning using a second data set, and the data in the second data set includes: the modal data of the object in the modality and the candidate same-type objects of the object.
[0009] Optionally, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: determining the proportion of the candidate same-type objects of each modality in the multiple modalities; and obtaining the candidate same-type objects of the object based on the proportion.
[0010] Optionally, the candidate same-item object and the second fused feature of the candidate same-item object are pre-stored in a database, and the database stores a plurality of objects, including the candidate same-item object.
[0011] Optionally, the method further includes: acquiring object information of the same item of the same item; and filling in the release information of the object according to the object information of the same item.
[0012] According to a second aspect of an embodiment of the present invention, there is provided an object processing method, comprising: receiving description information of an object, wherein the description information comprises image information and text information of the object; inputting the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a third data set, and data in the third data set comprises: the description information of the object and objects identical to the object.
[0013] Optionally, the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same object as the object, including: extracting image features of the image of the object and text features of the text of the object by the feature extraction module; fusing the image features and the text features by the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of a candidate same object of the object by the feature matching module to obtain the same object as the object.
[0014] Optionally, the multimodal model also includes: an image sub-model, wherein, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: inputting the image of the object into the image sub-model to obtain the first candidate same-type object of the object, wherein the image sub-model is obtained through machine learning using a fourth data set, and the data in the fourth data set includes: the image of the object and the first candidate same-type object of the object.
[0015] Optionally, the multimodal model also includes: a text sub-model, before matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: inputting the text of the object into the text sub-model to obtain the second candidate same object of the object, wherein the image sub-model is obtained through machine learning using a fifth data set, and the data in the fifth data set includes: the text of the object and the second candidate same object of the object.
[0016] Optionally, before matching the first fused feature with the second fused feature of a candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: when the candidate same object includes a first candidate same object and a second candidate same object, determining a ratio of the first candidate same object to the second candidate same object, wherein the first candidate same object is a candidate same object determined based on the image of the object, and the second candidate same object is a candidate same object determined based on the text of the object; and obtaining the candidate same object of the object according to the ratio.
[0017] Optionally, the candidate same-item object and the second fused feature of the candidate same-item object are pre-stored in a database, and the database stores a plurality of objects, including the candidate same-item object.
[0018] Optionally, the method further includes: acquiring object information of the same item of the same item; and filling in the release information of the object according to the object information of the same item.
[0019] According to a third aspect of an embodiment of the present invention, there is provided an object processing method, comprising: collecting a training data set, wherein the training data in the training data set comprises: description information of an object and objects of the same type as the object, wherein the description information comprises multimodal information of the object; and obtaining a multimodal model through machine training using the training data set.
[0020] According to a fourth aspect of an embodiment of the present invention, there is provided an object processing method, comprising: displaying an input box on an interactive interface, wherein the input box is used to input descriptive information of an object, wherein the descriptive information includes multimodal information of the object; in response to the input of the input box, displaying objects identical to the object on the interactive interface, wherein the objects identical to the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: descriptive information of the object and objects identical to the object.
[0021] According to a fifth aspect of an embodiment of the present invention, there is provided an object processing device, comprising: a first receiving module, for receiving description information of an object, wherein the description information includes multimodal information of the object; a first processing module, for inputting the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a first data set, and the data in the first data set includes: description information of the object and objects identical to the object.
[0022] According to a sixth aspect of an embodiment of the present invention, there is provided an object processing device, comprising: a second receiving module, for receiving description information of an object, wherein the description information comprises image information and text information of the object; a second processing module, for inputting the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a third data set, and the data in the third data set comprises: the description information of the object and objects identical to the object.
[0023] According to a seventh aspect of an embodiment of the present invention, there is provided an object processing device, comprising: an acquisition module, for acquiring a training data set, wherein the training data in the training data set comprises: description information of an object and objects of the same type as the object, wherein the description information comprises multimodal information of the object; and a training module, for obtaining a multimodal model through machine training using the training data set.
[0024] According to an eighth aspect of an embodiment of the present invention, there is provided an object processing device, comprising: a first display module, for displaying an input box on an interactive interface, wherein the input box is used to input descriptive information of an object, wherein the descriptive information includes multimodal information of the object; a second display module, for responding to input in the input box, and displaying objects of the same type as the object on the interactive interface, wherein the objects of the same type as the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: descriptive information of the object and objects of the same type as the object.
[0025] According to a ninth aspect of an embodiment of the present invention, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the object processing methods described above.
[0026] According to a tenth aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program; the processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes any one of the object processing methods described above.
[0027] According to an eleventh aspect of an embodiment of the present invention, there is provided an object processing method, comprising: receiving an image of an object and title information of the object; determining a first candidate same-type object based on the image of the object, and determining a second candidate same-type object based on the title information of the object; displaying a first sorting result of the first candidate same-type object, and a second sorting result of the second candidate same-type object; and highlighting the same-type object of the object based on the first sorting result and the second sorting result.
[0028] According to a twelfth aspect of an embodiment of the present invention, there is provided an object processing device, comprising: a third receiving module, for receiving an image of an object and title information of the object; a determination module, for determining a first candidate same-type object according to the image of the object, and for determining a second candidate same-type object according to the title information of the object; a display module, for displaying a first sorting result of the first candidate same-type object, and a second sorting result of the second candidate same-type object; a third display module, for highlighting the same-type object of the object according to the first sorting result and the second sorting result.
[0029] In an embodiment of the present invention, a method of receiving multimodal information of an object is adopted, and the multimodal information is processed through a multimodal model to obtain the same-item object of the object, thereby achieving the purpose of combining the multimodal information to determine the same-item object of the object, thereby achieving the technical effect of expanding the coverage of the same item and improving the same item recognition accuracy when searching for the same item, and further solving the technical problems of small coverage of searching for the same item and low same item recognition accuracy in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0031] Figure 1 A hardware structure block diagram of a computer terminal for implementing an object processing method is shown;
[0032] Figure 2 is a flowchart of an object processing method 1 according to embodiment 1 of the present invention;
[0033] Figure 3 is a flowchart of a second object processing method according to embodiment 1 of the present invention;
[0034] Figure 4 is a flowchart of a third object processing method according to embodiment 1 of the present invention;
[0035] Figure 5 is a flowchart of a fourth object processing method according to embodiment 1 of the present invention;
[0036] Figure 6 is a flowchart of a fifth object processing method according to embodiment 1 of the present invention;
[0037] Figure 7 is a schematic diagram of an object processing method according to an optional embodiment of the present invention;
[0038] Figure 8 is a structural block diagram of an object processing device 1 according to Embodiment 2 of the present invention;
[0039] Fig. 9 is a structural block diagram of an object processing device 2 according to embodiment 2 of the present invention;
[0040] Fig.10 is a structural block diagram of an object processing device 3 according to embodiment 2 of the present invention;
[0041] Fig.11 is a structural block diagram of an object processing device 4 according to Embodiment 2 of the present invention;
[0042] Fig.12 is a flow chart of an object processing device 5 according to embodiment 2 of the present invention;
[0043] Fig.13 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0046] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:
[0047] Deep Learning: Deep learning refers to a collection of algorithms that use various machine learning algorithms on multi-layer neural networks to solve various problems such as images and texts. Deep learning can be classified as neural networks in general, but there are many changes in specific implementations. The core of deep learning is feature learning, which aims to obtain hierarchical feature information through a layered network, thereby solving important problems that previously required artificially designed features.
[0048] Multimodality: The original meaning of multimodality used in multimodal biometrics refers to the integration or fusion of two or more biometric technologies, using the unique advantages of multiple biometric technologies and combining data fusion technology to make the authentication and identification process more accurate and secure. In the embodiment of the present invention, multimodality means information that represents an object from multiple perspectives. For example, multimodality can include: images, text, voice, video, animation, etc.
[0049] Example 1
[0050] According to an embodiment of the present invention, a method embodiment of an object processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0051] The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an object processing method. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0052] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0053] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the object processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the vulnerability detection method of the above-mentioned application program. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0054] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.
[0055] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).
[0056] Under the above operating environment, this application provides Figure 2 The object processing method shown. Figure 2 is a flowchart of an object processing method 1 according to Embodiment 1 of the present invention, and the process includes the following steps:
[0057] Step S202, receiving description information of the object, wherein the description information includes multimodal information of the object;
[0058] Step S204: input the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained by machine learning using a first data set, and the data in the first data set includes: description information of the object and objects identical to the object.
[0059] Through the above steps, the method of receiving multimodal information of the object is adopted, and the multimodal information is processed through the multimodal model to obtain the same object of the object, thereby achieving the purpose of combining the multimodal information to determine the same object of the object, thereby achieving the technical effect of expanding the coverage of the same item and improving the accuracy of identifying the same item when searching for the same item, and further solving the technical problems of small coverage of searching for the same item and low accuracy of identifying the same item in related technologies.
[0060] As an optional embodiment, the execution subject of the above method can be any computing device that can be used to perform calculations, for example, it can be an independent computer terminal, a computer cluster with strong computing power, or a server deployed with computing tasks.
[0061] As an optional embodiment, the above-mentioned object may refer to the content to be searched, that is, the standard for searching for the same item. The object may be in various forms, for example, it may be a specific commodity or a virtual service.
[0062] As an optional embodiment, the same type of object as the object referred to above, that is, the same type of object and the object belonging to the same type may need to meet certain same type conditions. Whether the two are the same type can be determined by training the multimodal model based on the accuracy requirements of the specific search. For example, when the search accuracy requirements are high, the same type condition may be relatively high, that is, the training of the multimodal model is more demanding; and when the search accuracy requirements are relatively general, the same type condition may be relatively low, so the training of the multimodal model is relatively loose.
[0063] For example, when the object is a specific commodity, when judging whether two commodities are the same, the following situations may be considered the same: 1. The main commodities are exactly the same; 2. The main commodities are basically the same, and it is impossible to determine whether the brands are the same through pictures or titles, or the brands are different but not the key attributes; 3. The main commodities are the same, but the colors are different (except for beauty products, food, and beverages); 4. The main commodities are the same, with the same patterns, but different directions; 5. The main commodities are the same, but the specifications are different (milliliters, number of packages, etc.); 6. The main commodities are the same, but the outer packaging and inner packaging are different; 7. The main commodities are pets or plants of the same variety; 8. The main commodities are bulk foods in non-standard packaging with the same appearance; 9. The main commodities have multiple themes (colors, patterns, etc., such as a series of mobile phone cases, a series of T-shirts), and belong to the same series; 10. The main commodities are inconsistent, but they are the same according to the titles. The cases where they are not the same items include: 1. Judging from the pictures and titles, they do not meet the same item requirements; 2. The product categories are different; 3. The brands of the products are different, which can be distinguished through pictures or titles, and the brand is the key attribute (such as branded products); 4. The colors and functions of beauty, food, and beverage products are different.
[0064] As an optional embodiment, the multimodal information of the object may include multiple types, for example, may include at least one of the following: image, text, video, voice, animation.
[0065] As an optional embodiment, the structure of the multimodal model can be diverse. For example, the structure of the multimodal model can include: a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same object of the object, including: extracting the features of the object in each mode through the feature extraction module; fusing the features in each mode through the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object. Since the first fused feature fuses the features of each mode of the object in multimodality, that is, the multimodal model can not only capture the intra-modal features within each individual mode, but also capture the inter-modal features between each mode and other modes. Therefore, the multimodal model obtained by training the fused features can make the precision of the searched object same style higher, that is, the accuracy is higher.
[0066] As an optional embodiment, in order to further improve the accuracy of the multimodal model in searching for the same object, when obtaining the candidate same object, it can also be obtained by means of an artificial intelligence algorithm model. For example, the multimodal model can also include: a modal sub-model, before matching the first fusion feature with the second fusion feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: inputting the modal data of the object in the modality into the modal sub-model to obtain the candidate same object of the object, wherein the modal sub-model is obtained by machine learning using a second data set, and the data in the second data set includes: the modal data of the object in the modality and the candidate same object of the object. It should be noted that the number of the above-mentioned modal sub-models can be multiple. For example, corresponding modal sub-models can be set for different modalities. For example, when the modality is image, the modal sub-model can be an image sub-model for searching for objects of the same type based on images; when the modality is text, the modal sub-model can be a text sub-model for searching for objects of the same type based on text; when the modality is voice, the modal sub-model can be a voice sub-model for searching for objects of the same type based on voice; when the modality is video, the modal sub-model can be a video sub-model for searching for objects of the same type based on videos; when the modality is animation, the modal sub-model can be an animation sub-model for searching for objects of the same type based on animations; and so on.
[0067] As an optional embodiment, before matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: determining the proportion of the candidate same objects of each modality in multiple modalities; obtaining the candidate same objects of the object according to the proportion. For example, when the multimodality includes images and texts, the proportion can be: the ratio between the number of candidate same objects searched based on images and the number of candidate same objects searched based on texts; when the multimodality includes images, texts and videos, the proportion can be: the ratio between the number of candidate same objects searched based on images, the number of candidate same objects searched based on texts, and the number of candidate same objects searched based on videos; and so on.
[0068] As an optional embodiment, in order to improve the efficiency of multimodal model searching for the same object, the candidate same object and the second fusion feature of the candidate same object can be pre-stored in a database, and the database stores multiple objects, including the candidate same object. By pre-storing the candidate same object and the second fusion feature of the candidate same object in the database, after extracting the first fusion feature of the object, the first fusion feature and the second fusion feature can be directly matched and compared to obtain the corresponding candidate same object, which can effectively save the time of temporarily extracting the second fusion feature of the candidate same object, thereby improving the speed of multimodal model searching for the same object and improving the efficiency of searching and recalling the same object.
[0069] As an optional embodiment, after obtaining the same object as the object, the same object information of the same object can also be obtained; and the release information of the object is filled in according to the same object information. For example, when a seller releases a product, he needs to upload a product picture and briefly fill in some product information. Therefore, as long as the same product of the product is intelligently searched through the above multimodal model, the product can be automatically marked with detailed and accurate product information, simplifying the process of product release, and the rich structured information can promote the search and recall of the product.
[0070] Figure 3 is a flowchart of a second object processing method according to Embodiment 1 of the present invention, and the flow includes the following steps:
[0071] Step S302, receiving description information of the object, wherein the description information includes image information and text information of the object;
[0072] Step S304: input the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained by machine learning using a third data set, and the data in the third data set includes: description information of the object and objects identical to the object.
[0073] Through the above steps, the method of receiving the image information and text information of the object is adopted, and the image information and text information are processed by the multimodal model to obtain the same object of the object, thereby achieving the purpose of combining the image information and the text information to determine the same object of the object, thereby achieving the technical effect of expanding the coverage of the same item and improving the recognition accuracy of the same item when searching for the same item, and further solving the technical problems of small coverage of searching for the same item and low recognition accuracy of the same item in related technologies.
[0074] As an optional embodiment, the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same type of object, including: extracting the image features of the image of the object and the text features of the text of the object by the feature extraction module; fusing the image features and the text features by the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same type of object by the feature matching module to obtain the same type of object. Since the first fused feature fuses the image features and the text features of the object, that is, the multimodal model can not only capture the image features of the image and the text features of the text, but also capture the associated features between the image and the text, that is, the associated relationship between the image features and the text features, therefore, the multimodal model obtained by training the fused features can make the precision of the searched object same type higher, that is, the accuracy is higher.
[0075] As an optional embodiment, in order to further improve the accuracy of the multimodal model in searching for the same object, when obtaining the candidate same object, it can also be obtained by means of an artificial intelligence algorithm model. For example, the multimodal model can also include: an image sub-model, wherein, before matching the first fusion feature with the second fusion feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: inputting the image of the object into the image sub-model to obtain the first candidate same object of the object, wherein the image sub-model is obtained by machine learning using a fourth data set, and the data in the fourth data set includes: the image of the object and the first candidate same object of the object.
[0076] As an optional embodiment, the multimodal model may further include: a text sub-model, before matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: inputting the text of the object into the text sub-model to obtain the second candidate same object of the object, wherein the image sub-model is obtained through machine learning using the fifth data set, and the data in the fifth data set includes: the text of the object and the second candidate same object of the object.
[0077] It should be noted that the use of the above-mentioned image sub-model to obtain the first candidate same object and the use of the text sub-model to obtain the second candidate same object can improve the efficiency of obtaining candidate same objects. Since the above-mentioned image sub-model and text sub-model are also obtained through training with a large amount of training data, they can also effectively improve the accuracy of obtaining candidate same objects to a certain extent, thereby indirectly improving the efficiency and accuracy of obtaining same objects.
[0078] As an optional embodiment, before matching the first fusion feature with the second fusion feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: when the candidate same object includes the first candidate same object and the second candidate same object, determining the ratio of the first candidate same object to the second candidate same object, wherein the first candidate same object is the candidate same object determined based on the image of the object, and the second candidate same object is the candidate same object determined based on the text of the object; the candidate same object of the object is obtained according to the ratio. It should be noted that the above ratio can be flexibly configured according to the specific situation. For example, when the number of the first candidate same object determined by the image is significantly greater than the second candidate same object determined by the text, the ratio can be configured to be higher; and when the number of the first candidate same object determined by the image is significantly less than the second candidate same object determined by the text, the ratio can be configured to be lower.
[0079] As an optional embodiment, in order to improve the efficiency of the multimodal model in searching for the same object, the candidate same object and the second fusion feature of the candidate same object can be pre-stored in a database, and the database stores multiple objects, including the candidate same object.
[0080] As an optional embodiment, after obtaining the same object as the object, the same object information of the same object can also be obtained; and the object publishing information is filled in according to the same object information. In the above manner, the object publishing information is filled in directly according to the same object information, which can efficiently fill in the publishing information for the object, simplify the object publishing process and improve the publishing efficiency.
[0081] Figure 4 is a flowchart of a third object processing method according to Embodiment 1 of the present invention, and the process includes the following steps:
[0082] Step S402, collecting a training data set, wherein the training data in the training data set includes: description information of an object and objects of the same type as the object, wherein the description information includes multimodal information of the object;
[0083] Step S404: Using the training data set to perform machine training to obtain a multimodal model.
[0084] Through the above steps, a multimodal model is obtained by machine training using a training data set, and the purpose of training the model based on the multimodal information of the object is achieved, thereby providing a basis for intelligent search for the same item, and then being able to achieve the technical effect of expanding the coverage of the same item and improving the accuracy of identifying the same item when searching for the same item, thereby solving the technical problems of small coverage of searching for the same item and low accuracy of identifying the same item in related technologies.
[0085] Figure 5 is a flowchart of a fourth object processing method according to Embodiment 1 of the present invention, and the flow includes the following steps:
[0086] Step S502, displaying an input box on the interactive interface, wherein the input box is used to input description information of the object, wherein the description information includes information under multimodal state of the object;
[0087] Step S504, in response to the input in the input box, display the same type of objects as the object on the interactive interface, wherein the same type of objects as the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: description information of the object and the same type of objects as the object.
[0088] Through the above steps, the multimodal information of the object is displayed on the interactive interface, and the same object of the object obtained by using the multimodal model is displayed on the interactive interface, so as to achieve the purpose of obtaining the same object of the object according to the multimodal model, thereby achieving the technical effect of expanding the coverage of the same items and improving the accuracy of the same item recognition when searching for the same items, and further solving the technical problems of small coverage of the same item search and low accuracy of the same item recognition in the related technology.
[0089] Figure 6 is a flowchart of a fifth object processing method according to Embodiment 1 of the present invention, and the process includes the following steps:
[0090] Step S602, receiving an image of an object and title information of the object;
[0091] Step S604, determining a first candidate object of the same item according to the image of the object, and determining a second candidate object of the same item according to the title information of the object;
[0092] Step S606, displaying the first ranking result of the first candidate same-item object and the second ranking result of the second candidate same-item object;
[0093] Step S608: highlighting objects of the same type as the object according to the first sorting result and the second sorting result.
[0094] Through the above steps, a training image is used to determine the first candidate same object, and a second candidate same object is determined according to the title information of the object. By displaying the first sorting result of the first candidate same object and the second sorting result of the second candidate same object, and highlighting the same object of the object on the first sorting result and the second sorting result, the purpose of searching and highlighting the same object of the object based on the image and title information results is achieved, thereby achieving the technical effect of quickly and accurately determining the same object of the object, and further solving the technical problems of small search coverage and low same recognition accuracy in related technologies.
[0095] As an optional embodiment, when determining the first candidate same-item object based on the image of the object, and determining the second candidate same-item object based on the title information of the object, a variety of methods can be used. For example, it can be determined by an artificial intelligence model. For example, the image of the object is input into an image recall model to obtain the first candidate same-item object of the object; the title information of the object is input into a text recall model to obtain the second candidate same-item object of the object. The above-mentioned image recall model and text recall model are both deep neural network models, and can be obtained by training with a large amount of real data.
[0096] As an optional embodiment, when displaying the first ranking result of the first candidate object of the same paragraph and the second ranking result of the second candidate object of the same paragraph, the same ranking standard can be used when ranking the first candidate object of the same paragraph and the second candidate object of the same paragraph, or different ranking standards can be used. For example, when ranking the first candidate object of the same paragraph, the ranking can be based on the size of the similarity with the object, and when ranking the second candidate object of the same paragraph, the ranking can also be based on the size of the similarity with the object. For another example, when ranking the first candidate object of the same paragraph, the ranking can be based on the size of the similarity with the object, and when ranking the second candidate object of the same paragraph, the ranking can be based on the size of the association with the object.
[0097] As an optional embodiment, when highlighting the same object of the object according to the first sorting result and the second sorting result, multiple methods can also be used. For example, the ratio of the first candidate same object to the second candidate same object can be proportionally highlighted. For example, the number of the first candidate same object determined according to the image is 10, and the number of the second candidate same object determined according to the title information is 5; when the number of the same object of the object to be highlighted is 6, the number of the first candidate same object highlighted is 4, and the number of the second candidate same object highlighted is 2. In addition, there are also multiple ways to highlight the same object of the object, for example, it can be achieved by marking the first candidate same object and the second candidate same object to be highlighted, it can also be achieved by adding background color to the first candidate same object and the second candidate same object to be highlighted, it can also be achieved by adding animation to the first candidate same object and the second candidate same object to be highlighted, etc. It should be noted that the above-mentioned highlighting method is only an example. As long as the processing method can reflect the difference between the first candidate object of the same item and the second candidate object of the same item that do not need to be highlighted, it belongs to the embodiment of the present invention and will not be exemplified one by one here.
[0098] In combination with the above embodiment and the optional embodiment, an optional implementation is provided. It should be noted that in this optional implementation, a commodity is taken as an object, and the image and text of the commodity are taken as the multimodal information of the commodity for illustration.
[0099] In C2C e-commerce platforms, ordinary sellers have not received similar professional training as professional sellers on B2C platforms. When publishing products, the product description information is incomplete, the product information filled in by the editor is limited, and the product information is often filled in incorrectly or omitted, resulting in the inability of the product to be effectively recalled by the search engine, forming a transaction barrier between the seller and the buyer, which has a significant negative impact on the buyer's search satisfaction and the seller's selling efficiency. In this optional implementation, a multimodal product same-item search algorithm is used to enrich product information. For example, product images and title information (because product images and titles generally contain key product information) are used to search for the same product through a multimodal product same-item search model, thereby improving the recall efficiency of the search process and increasing the product transaction rate.
[0100] In related technologies, when searching for the same product based on the product, the product image is generally used, and the trained deep learning network is used to search for the corresponding same product. The input is the image of the product, which is processed by the deep learning network to obtain the encoding vector, and finally the same product is obtained by searching in the library (for example, the product database), and the final result is output by means of the same score card threshold. However, when searching for the same product using the above method, only the image information is used, resulting in low coverage and limited help in the final product search recall.
[0101] For example: 1) Insufficient coverage of same-style signatures; because the same-style search algorithm in the relevant technology only uses image information, in the C2C e-commerce platform, a large number of pictures uploaded by sellers are irrelevant to the products, such as emoticons and product packaging boxes, which are difficult to recall after passing the card threshold, resulting in low coverage; 2) Low accuracy; The same-style search algorithm in the relevant technology only uses image information, and some products between categories are difficult to distinguish by pure images, such as mobile phones and mobile phone films, resulting in limited algorithm accuracy for products in these categories.
[0102] In view of this, in this optional implementation, the image and title dual-path recall of candidate identical products is introduced to train a multimodal model of images and texts with multimodal feature refinement. The image and title dual-path recall increases the coverage of products, and the multimodal features utilize image and title information to effectively improve the accuracy of identical product recognition.
[0103] Figure 7 is a schematic diagram of an object processing method according to an optional embodiment of the present invention, such as Figure 7 As shown, the object processing method includes the following parts:
[0104] (1) Image recall model (in the figure, it is called image recall, and its function is the same as the above image sub-model). The image recall model is also a deep convolutional network. The training data for this part is image pairs. In many e-commerce scenarios, there is a wealth of click pair data. The above image recall model can be obtained through training. In actual application scenarios, when an e-commerce product image is input, the corresponding candidate products of the same item will be recalled.
[0105] (2) Text recall model (shown as text recall in the figure, with the same function as the above text sub-model). The text recall model is an algorithm based on text search, which can obtain candidate identical products based on the product title.
[0106] (3) Image-text multimodal model (same function as the above multimodal model). The image-text multimodal model is a neural network model based on image convolutional neural networks (CNN) and text-based deep semantic models (DSSM). The training data source is the return data of Xianyu. The encoding vectors of images and texts are extracted respectively, and then the fusion module is used to perform feature fusion to obtain multimodal feature expression. In actual use, this feature is used to sort the above two recalled products to obtain the final associated identical items.
[0107] Through the above optional implementation methods, in view of the coverage and accuracy issues of the same-item search algorithm in the related technology, a solution for multimodal product same-item search using product images and titles is proposed. In this copy, the reflux data system is used, the product images and titles are recalled in two ways, and the multimodal feature sorting of pictures and texts is used to search for the same item, which not only expands the coverage of the same item, but also ensures the accuracy of the same item recognition.
[0108] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.
[0110] Example 2
[0111] According to an embodiment of the present invention, a device for implementing the above-mentioned object processing method 1 is also provided. Figure 8 is a structural block diagram of an object processing device 1 according to Embodiment 2 of the present invention. Figure 8 As shown, the device includes: a first receiving module 82 and a first processing module 84. The device is described below.
[0112] The first receiving module 82 is used to receive descriptive information of an object, wherein the descriptive information includes multimodal information of the object; the first processing module 84 is connected to the first receiving module 82, and is used to input the descriptive information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a first data set, and the data in the first data set includes: descriptive information of the object and objects identical to the object.
[0113] It should be noted that the first receiving module 82 and the first processing module 84 correspond to steps S202 to S204 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.
[0114] According to an embodiment of the present invention, a device for implementing the second object processing method is also provided. Fig. 9 is a structural block diagram of an object processing device 2 according to embodiment 2 of the present invention. Fig. 9 As shown, the device includes: a second receiving module 92 and a second processing module 94. The device is described below.
[0115] The second receiving module 92 is used to receive descriptive information of the object, wherein the descriptive information includes image information and text information of the object; the second processing module 94 is connected to the second receiving module 92, and is used to input the descriptive information into the multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a third data set, and the data in the third data set includes: descriptive information of the object and objects identical to the object.
[0116] It should be noted that the second receiving module 92 and the second processing module 94 correspond to steps S302 to S304 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.
[0117] According to an embodiment of the present invention, a device for implementing the above-mentioned object processing method 3 is also provided. Fig.10 is a structural block diagram of an object processing device 3 according to Embodiment 2 of the present invention. Fig.10 As shown, the device includes: a collection module 102 and a training module 104. The device is described below.
[0118] The acquisition module 102 is used to acquire a training data set, wherein the training data in the training data set includes: description information of an object and objects of the same type as the object, wherein the description information includes multimodal information of the object; the training module 104 is connected to the above-mentioned acquisition module 102, and is used to use the training data set to obtain a multimodal model through machine training.
[0119] It should be noted that the acquisition module 102 and the training module 104 correspond to steps S402 to S404 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.
[0120] According to an embodiment of the present invention, a device for implementing the fourth object processing method is also provided. Fig.11 is a structural block diagram of an object processing device 4 according to Embodiment 2 of the present invention. Fig.11 As shown, the device includes: a first display module 112 and a second display module 114. The device is described below.
[0121] The first display module 112 is used to display an input box on the interactive interface, wherein the input box is used to input descriptive information of the object, wherein the descriptive information includes multimodal information of the object; the second display module 114 is connected to the first display module 112, and is used to respond to input in the input box and display objects of the same type as the object on the interactive interface, wherein the objects of the same type as the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: descriptive information of the object and objects of the same type as the object.
[0122] It should be noted that the first display module 112 and the second display module 114 correspond to steps S502 to S504 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.
[0123] According to an embodiment of the present invention, a device for implementing the fifth object processing method is also provided. Fig.12 is a structural block diagram of an object processing device 5 according to Embodiment 2 of the present invention. Fig.12 As shown, the device includes: a third receiving module 122, a determining module 124, a display module 126 and a third display module 128. The device is described below.
[0124] A third receiving module 122 is used to receive an image of an object and title information of the object; a determination module 124 is connected to the third receiving module 122, and is used to determine a first candidate same-type object based on the image of the object, and to determine a second candidate same-type object based on the title information of the object; a display module 126 is connected to the determination module 124, and is used to display a first sorting result of the first candidate same-type object, and a second sorting result of the second candidate same-type object; a third display module 128 is connected to the display module 126, and is used to highlight the same-type object of the object based on the first sorting result and the second sorting result.
[0125] It should be noted that the third receiving module 122, the determining module 124, the display module 126 and the third display module 128 correspond to steps S602 to S604 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.
[0126] Example 3
[0127] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.
[0128] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.
[0129] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the vulnerability detection method of the application: receiving description information of an object, wherein the description information includes information under the multimodality of the object; inputting the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a first data set, and the data in the first data set includes: description information of the object and objects identical to the object.
[0130] Optionally, Fig.13 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Fig.13 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 132, a memory 134, etc.
[0131] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the security vulnerability detection method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned detection method of system vulnerability attacks. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0132] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive description information of the object, wherein the description information includes information under the multimodality of the object; input the description information into the multimodal model to obtain the same type of objects as the object, wherein the multimodal model is obtained through machine learning using a first data set, and the data in the first data set includes: the description information of the object and the same type of objects as the object.
[0133] Optionally, the processor may also execute program code of the following steps: the multimodal information of the object includes at least one of the following: image, text, video, voice, and animation.
[0134] Optionally, the processor may also execute the program code of the following steps: the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same-type object of the object, including: extracting the features of the object in each modality through the feature extraction module; fusing the features in each modality through the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object.
[0135] Optionally, the processor may also execute the program code of the following steps: the multimodal model also includes: a modal sub-model, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: inputting the modal data of the object under the modality into the modal sub-model to obtain the candidate same-type object of the object, wherein the modal sub-model is obtained through machine learning using a second data set, and the data in the second data set includes: the modal data of the object under the modality and the candidate same-type objects of the object.
[0136] Optionally, the processor may also execute the program code of the following steps: before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: determining the proportion of the candidate same-type objects of each modality in multiple modalities; and obtaining the candidate same-type objects of the object based on the proportion.
[0137] Optionally, the processor may also execute program code of the following steps: the candidate same-item object and the second fusion feature of the candidate same-item object are pre-stored in a database, and the database stores a plurality of objects, including the candidate same-item object.
[0138] Optionally, the processor may also execute the program code of the following steps: obtaining the same-item object information of the same-item object; and filling in the object publishing information according to the same-item object information.
[0139] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive description information of the object, wherein the description information includes image information and text information of the object; input the description information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a third data set, and the data in the third data set includes: description information of the object and objects identical to the object.
[0140] Optionally, the processor may also execute the program code of the following steps: the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same object as the object, including: extracting the image features of the image of the object and the text features of the text of the object through the feature extraction module; fusing the image features and the text features through the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object as the object.
[0141] Optionally, the processor may also execute the program code of the following steps: the multimodal model also includes: an image sub-model, wherein, before matching the first fusion feature with the second fusion feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: inputting the image of the object into the image sub-model to obtain the first candidate same-type object of the object, wherein the image sub-model is obtained through machine learning using a fourth data set, and the data in the fourth data set includes: the image of the object and the first candidate same-type object of the object.
[0142] Optionally, the processor may also execute the program code of the following steps: the multimodal model also includes: a text sub-model, before matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: inputting the text of the object into the text sub-model to obtain the second candidate same object of the object, wherein the image sub-model is obtained through machine learning using the fifth data set, and the data in the fifth data set includes: the text of the object and the second candidate same object of the object.
[0143] Optionally, the processor may also execute the program code of the following steps: before matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: when the candidate same object includes the first candidate same object and the second candidate same object, determining the ratio of the first candidate same object to the second candidate same object, wherein the first candidate same object is a candidate same object determined based on the image of the object, and the second candidate same object is a candidate same object determined based on the text of the object; and obtaining the candidate same object of the object according to the ratio.
[0144] Optionally, the processor may also execute program code of the following steps: the candidate same-item object and the second fusion feature of the candidate same-item object are pre-stored in a database, and the database stores a plurality of objects, including the candidate same-item object.
[0145] Optionally, the processor may also execute the program code of the following steps: obtaining the same-item object information of the same-item object; and filling in the object publishing information according to the same-item object information.
[0146] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collect a training data set, wherein the training data in the training data set includes: description information of the object and objects of the same type as the object, wherein the description information includes information under multimodal conditions of the object; use the training data set for machine training to obtain a multimodal model.
[0147] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: display an input box on the interactive interface, wherein the input box is used to input descriptive information of the object, wherein the descriptive information includes information under the multimodality of the object; in response to the input in the input box, display the same type of objects as the object on the interactive interface, wherein the same type of objects as the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: the descriptive information of the object and the same type of objects as the object.
[0148] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive the image of the object and the title information of the object; determine the first candidate same-type object based on the image of the object, and determine the second candidate same-type object based on the title information of the object; display the first sorting result of the first candidate same-type object and the second sorting result of the second candidate same-type object; highlight the same-type object of the object based on the first sorting result and the second sorting result.
[0149] The method of receiving multimodal information of an object is adopted, and the multimodal information is processed through a multimodal model to obtain the same object of the object, thereby achieving the purpose of combining the multimodal information to determine the same object of the object, thereby achieving the technical effect of expanding the coverage of the same items and improving the accuracy of identifying the same items when searching for the same items, and further solving the technical problems of small coverage of searching for the same items and low accuracy of identifying the same items in related technologies.
[0150] It can be understood by those skilled in the art that Fig.13 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.13 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Fig.13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.13 Different configurations shown.
[0151] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0152] Example 4
[0153] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the object processing method provided in the above embodiment 1.
[0154] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.
[0155] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving descriptive information of an object, wherein the descriptive information includes information under multimodality of the object; inputting the descriptive information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a first data set, and the data in the first data set includes: the descriptive information of the object and objects identical to the object.
[0156] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: the multimodal information of the object includes at least one of the following: image, text, video, voice, animation.
[0157] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same-type object of the object, including: extracting features of the object in each modality through the feature extraction module; fusing the features in each modality through the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object.
[0158] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: the multimodal model also includes: a modal sub-model, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: inputting the modal data of the object under the modality into the modal sub-model to obtain the candidate same-type object of the object, wherein the modal sub-model is obtained through machine learning using a second data set, and the data in the second data set includes: the modal data of the object under the modality and the candidate same-type objects of the object.
[0159] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: determining the proportion of candidate same-type objects of each modality in multiple modalities; and obtaining the candidate same-type objects of the object based on the proportion.
[0160] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: candidate same-item objects and second fusion features of the candidate same-item objects are pre-stored in a database, and the database stores multiple objects, including the candidate same-item objects.
[0161] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: obtaining the same item object information of the same item object; and filling in the object release information according to the same item object information.
[0162] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving descriptive information of an object, wherein the descriptive information includes image information and text information of the object; inputting the descriptive information into a multimodal model to obtain objects identical to the object, wherein the multimodal model is obtained through machine learning using a third data set, and the data in the third data set includes: the descriptive information of the object and objects identical to the object.
[0163] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: the multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same type of object as the object, including: extracting image features of the image of the object and text features of the text of the object through the feature extraction module; fusing the image features and the text features through the feature fusion module to obtain a first fused feature; matching the first fused feature with the second fused feature of the candidate same type of object of the object through the feature matching module to obtain the same type of object as the object.
[0164] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: the multimodal model also includes: an image sub-model, wherein, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: inputting the image of the object into the image sub-model to obtain the first candidate same-type object of the object, wherein the image sub-model is obtained through machine learning using a fourth data set, and the data in the fourth data set includes: the image of the object and the first candidate same-type object of the object.
[0165] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: the multimodal model also includes: a text sub-model, before matching the first fused feature with the second fused feature of the candidate same-type object of the object through the feature matching module to obtain the same-type object of the object, it also includes: inputting the text of the object into the text sub-model to obtain the second candidate same-type object of the object, wherein the image sub-model is obtained through machine learning using the fifth data set, and the data in the fifth data set includes: the text of the object and the second candidate same-type object of the object.
[0166] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: before matching the first fused feature with the second fused feature of the candidate same object of the object through the feature matching module to obtain the same object of the object, it also includes: when the candidate same object includes the first candidate same object and the second candidate same object, determining the ratio of the first candidate same object to the second candidate same object, wherein the first candidate same object is a candidate same object determined based on the image of the object, and the second candidate same object is a candidate same object determined based on the text of the object; and obtaining the candidate same object of the object according to the ratio.
[0167] Optionally, in this embodiment, the storage medium is also configured to store program code for executing the following steps: candidate same-item objects and second fusion features of the candidate same-item objects are pre-stored in a database, and the database stores multiple objects, including the candidate same-item objects.
[0168] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: obtaining the same item object information of the same item object; and filling in the object release information according to the same item object information.
[0169] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a training data set, wherein the training data in the training data set includes: descriptive information of an object and objects of the same type as the object, wherein the descriptive information includes multimodal information of the object; and obtaining a multimodal model through machine training using the training data set.
[0170] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: displaying an input box on an interactive interface, wherein the input box is used to input descriptive information of an object, wherein the descriptive information includes information under the object's multimodality; in response to the input in the input box, displaying objects similar to the object on the interactive interface, wherein the objects similar to the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: descriptive information of the object and objects similar to the object.
[0171] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: receiving an image of an object and title information of the object; determining a first candidate same-type object based on the image of the object, and determining a second candidate same-type object based on the title information of the object; displaying a first sorting result of the first candidate same-type object, and a second sorting result of the second candidate same-type object; and highlighting the same-type object of the object based on the first sorting result and the second sorting result.
[0172] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0173] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0174] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0176] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0178] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An object processing method, characterized in that: include: Receiving description information of an object, wherein the description information includes multimodal information of the object; Inputting the description information into a multimodal model to obtain objects of the same type as the object, wherein the multimodal model is obtained by machine learning using a first data set, and the data in the first data set includes: description information of the object and objects of the same type as the object; Among them, before the description information is input into the multimodal model to obtain the same object as the object, it includes: the multimodal model also includes: a modal sub-model, the modal data of the object in the modality is input into the modal sub-model to obtain candidate same objects as the object; the proportion of candidate same objects of each modality in multiple modalities is determined; and the candidate same objects of the object are obtained according to the proportion.
2. The method according to claim 1, characterized in that The multimodal information of the object includes at least one of the following: image, text, video, voice, and animation.
3. The method according to claim 1, characterized in that The multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same object as the object, including: Extracting features of the object in each mode by the feature extraction module; The features under the various modes are fused by the feature fusion module to obtain a first fused feature; The first fused feature is matched with the second fused feature of the candidate same-type object of the object by the feature matching module to obtain the same-type object of the object.
4. The method according to claim 1, characterized in that: The modal sub-model is obtained through machine learning using a second data set, where data in the second data set includes: modal data of an object in a modality and candidate identical objects of the object.
5. The method according to claim 3, characterized in that: The candidate same-type object and the second fused feature of the candidate same-type object are pre-stored in a database, and the database stores a plurality of objects, including the candidate same-type object.
6. The method according to any one of claims 1 to 5, characterized in that Also includes: Obtaining information about the same item of the same item; Fill in the release information of the object according to the information of the same object.
7. An object processing method, characterized in that: include: Receiving description information of an object, wherein the description information includes image information and text information of the object; Inputting the description information into a multimodal model to obtain an object identical to the object, wherein the multimodal model is obtained by machine learning using a third data set, and the data in the third data set includes: description information of the object and an object identical to the object; Wherein, before the description information is input into the multimodal model to obtain the same object of the object, it also includes: the multimodal model also includes: an image sub-model and a text sub-model, and when the candidate same objects of the object include a first candidate same object and a second candidate same object, a ratio of the first candidate same object to the second candidate same object is determined, wherein the first candidate same object is a candidate same object determined by using the image sub-model based on the image of the object, and the second candidate same object is a candidate same object determined by using the text sub-model based on the text of the object; the candidate same object of the object is obtained according to the ratio.
8. The method according to claim 7, characterized in that The multimodal model includes a feature extraction module, a feature fusion module and a feature matching module, wherein the description information is input into the multimodal model to obtain the same object as the object, including: Extracting image features of the image of the object and text features of the text of the object by the feature extraction module; The image feature and the text feature are fused by the feature fusion module to obtain a first fused feature; The first fused feature is matched with the second fused feature of the candidate same-type object of the object by the feature matching module to obtain the same-type object of the object.
9. The method according to claim 8, characterized in that Before matching the first fused feature with the second fused feature of the candidate same-item object of the object by the feature matching module to obtain the same-item object of the object, the method further includes: The image of the object is input into the image sub-model to obtain a first candidate same-type object of the object, wherein the image sub-model is obtained by machine learning using a fourth data set, and the data in the fourth data set includes: an image of the object and a first candidate same-type object of the object.
10. The method according to claim 8, characterized in that Before matching the first fused feature with the second fused feature of the candidate same-item object of the object by the feature matching module to obtain the same-item object of the object, the method further includes: The text of the object is input into the text sub-model to obtain a second candidate identical object of the object, wherein the image sub-model is obtained by machine learning using a fifth data set, and the data in the fifth data set includes: the text of the object and the second candidate identical object of the object.
11. The method according to claim 8, characterized in that The candidate same-type object and the second fused feature of the candidate same-type object are pre-stored in a database, and the database stores a plurality of objects, including the candidate same-type object.
12. The method according to any one of claims 7 to 11, characterized in that Also includes: Obtaining information about the same item of the same item; Fill in the release information of the object according to the information of the same object.
13. The method according to any one of claims 7 to 11, characterized in that The text information includes title information, and the method further includes: Displaying a first ranking result of the first candidate same-item object and a second ranking result of the second candidate same-item object; According to the first sorting result and the second sorting result, objects of the same item as the object are highlighted.
14. An object processing method, characterized in that: include: Collecting a training data set, wherein the training data in the training data set includes: description information of an object and objects of the same type as the object, wherein the description information includes multimodal information of the object; Using the training data set to obtain a multimodal model through machine training; Wherein, the multimodal model includes: a modal sub-model, wherein the modal sub-model is used to obtain candidate same-type objects of the object based on the modal data of the object in the modality, wherein the multimodal model is used to obtain candidate same-type objects of the object according to the proportion of candidate same-type objects of each modality in multiple modalities.
15. An object processing method, characterized in that: include: Displaying an input box on the interactive interface, wherein the input box is used to input description information of the object, wherein the description information includes information of the object in multiple modes; In response to the input in the input box, displaying the same objects as the object on the interactive interface, wherein the same objects as the object are obtained through a multimodal model, and the multimodal model is obtained through machine learning using a sixth data set, and the data in the sixth data set includes: description information of the object and the same objects as the object; Wherein, the multimodal model includes: a modal sub-model, wherein the modal sub-model is used to obtain candidate same-type objects of the object based on the modal data of the object in the modality, wherein the multimodal model is used to obtain candidate same-type objects of the object according to the proportion of candidate same-type objects of each modality in multiple modalities.
16. An object processing device, characterized in that: include: A first receiving module, configured to receive description information of an object, wherein the description information includes multimodal information of the object; A first processing module, configured to input the description information into a multimodal model to obtain objects of the same type as the object, wherein the multimodal model is obtained by machine learning using a first data set, and the data in the first data set includes: description information of the object and objects of the same type as the object; Among them, the multimodal model also includes: a modal sub-model, and the first processing module is also used to input the modal data of the object in the modality into the modal sub-model to obtain candidate same-type objects of the object; determine the proportion of candidate same-type objects of each modality in multiple modalities; and obtain candidate same-type objects of the object based on the proportion.
17. An object processing device, characterized in that: include: A second receiving module, configured to receive description information of an object, wherein the description information includes image information and text information of the object; A second processing module, configured to input the description information into a multimodal model to obtain objects of the same type as the object, wherein the multimodal model is obtained by machine learning using a third data set, and the data in the third data set includes: description information of the object and objects of the same type as the object; Wherein, the multimodal model also includes: an image sub-model and a text sub-model, and the second processing module is also used to determine the ratio of the first candidate same object and the second candidate same object when the candidate same object of the object includes a first candidate same object and a second candidate same object, wherein the first candidate same object is a candidate same object determined by using the image sub-model based on the image of the object, and the second candidate same object is a candidate same object determined by using the text sub-model based on the text of the object; the candidate same object of the object is obtained according to the ratio.
18. An object processing device, characterized in that: include: A collection module, used to collect a training data set, wherein the training data in the training data set includes: description information of an object and objects of the same type as the object, wherein the description information includes information of the object in multiple modes; A training module, used to obtain a multimodal model by machine training using the training data set; Wherein, the multimodal model includes: a modal sub-model, wherein the modal sub-model is used to obtain candidate same-type objects of the object based on the modal data of the object in the modality, wherein the multimodal model is used to obtain candidate same-type objects of the object according to the proportion of candidate same-type objects of each modality in multiple modalities.
19. An object processing device, characterized in that: include: A first display module is used to display an input box on the interactive interface, wherein the input box is used to input description information of the object, wherein the description information includes information of the object in multiple modes; A second display module, for responding to the input of the input box, and displaying the same object of the object on the interactive interface, wherein the same object of the object is obtained by a multimodal model, and the multimodal model is obtained by machine learning using a sixth data set, and the data in the sixth data set includes: description information of the object and the same object of the object; Wherein, the multimodal model includes: a modal sub-model, wherein the modal sub-model is used to obtain candidate same-type objects of the object based on the modal data of the object in the modality, wherein the multimodal model is used to obtain candidate same-type objects of the object according to the proportion of candidate same-type objects of each modality in multiple modalities.
20. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the object processing method according to any one of claims 1 to 15.
21. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the object processing method according to any one of claims 1 to 15.
