Product appearance compliance detection method and system based on multi-modal fusion

Abstract

The invention provides a multi-modal fusion product appearance compliance detection method and system, and the method comprises the steps: obtaining, by a terminal device, an appearance picture and text description information of a to-be-put-on-shelf product uploaded by a user, and sending an appearance compliance detection request message to a server after detecting a triggering operation on an appearance detection control; receiving at least one similar appearance design from the server; displaying at least one similar appearance design and a corresponding similarity score and feature comparison control on a product appearance compliance detection interface; detecting a trigger operation on any feature comparison control, and sending a feature comparison request message to a server; receiving a feature comparison table from the server; and displaying the feature comparison table on a product appearance compliance detection interface. Therefore, multi-modal fusion retrieval is carried out through an image-text fusion technology, a plurality of similar appearance designs and a feature comparison table with the to-be-put-on-shelf product are obtained, and the comprehensiveness and accuracy of product appearance compliance detection are improved.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a multi-modal fusion product appearance compliance detection method and system. Background Art

[0002] In compliance scenarios, compliance detection between appearance patents and products to be put on the shelves is a key task to ensure the appearance compliance of products to be put on the shelves. Traditional matching methods mainly rely on text descriptions, but since most of the information in appearance patents is presented in the form of images, the text descriptions are often short, resulting in limited matching results. The application of natural language processing (NLP) technology in such scenarios is limited. Summary of the invention

[0003] The present application provides a multimodal fusion product appearance compliance detection method and system, which performs multimodal fusion retrieval through image-text fusion technology to obtain multiple similar appearance designs and feature comparison tables with the products to be put on the shelves, significantly improving the effect of matching appearance patents with commodities, and improving the comprehensiveness and accuracy of product appearance compliance detection.

[0004] In a first aspect, the present application provides a product appearance compliance detection method based on multimodal fusion, the method being applied to a terminal device of an e-commerce platform, the e-commerce platform also including a server, the method including: obtaining appearance pictures and text description information of products to be listed uploaded by users, and displaying an appearance detection control on a product appearance compliance detection interface; and, detecting a triggering operation on the appearance detection control, creating an appearance compliance detection request message, the appearance compliance detection request message including the appearance picture and the text description information; and, sending the appearance compliance detection request message to the server, the appearance compliance detection request message being used to instruct the server to filter out at least one similar appearance design whose similarity to the product to be listed is greater than a preset similarity from an appearance design database, the appearance design database including appearance pictures and text description information of multiple appearance designs corresponding to the product category of the product to be listed, the similarity being comprehensively determined by the appearance pictures and text description information; An appearance compliance detection response message is received from the server, the appearance compliance detection response message includes the at least one similar appearance design; and the at least one similar appearance design, the similarity score of each similar appearance design, and the feature comparison control corresponding to each similar appearance design are displayed on the product appearance compliance detection interface; a triggering operation of a feature comparison control of any reference similar appearance design among the at least one similar appearance design is detected, and a feature comparison request message is created; and the feature comparison request message is sent to the server, the feature comparison request message is used to instruct the server to generate a feature comparison table of the product to be listed and the reference similar appearance design; a feature comparison response message is received from the server, the feature comparison response message includes the feature comparison table; and the feature comparison table is displayed on the product appearance compliance detection interface.

[0005] In a second aspect, the present application provides an e-commerce platform system, which includes a terminal device and a server, and the terminal device is used to execute the step instructions in the method as described in any one of the first aspects.

[0006] It can be seen that in the embodiment of the present application, the terminal device obtains the appearance picture and text description information of the product to be put on the shelf uploaded by the user, and displays the appearance detection control on the product appearance compliance detection interface; and, upon detecting the triggering operation of the appearance detection control, creates an appearance compliance detection request message, the appearance compliance detection request message includes the appearance picture and text description information; and, sends the appearance compliance detection request message to the server, the appearance compliance detection request message is used to instruct the server to filter out at least one similar appearance design whose similarity with the product to be put on the shelf is greater than a preset similarity from the appearance design database, the appearance design database includes the appearance pictures and text description information of multiple appearance designs corresponding to the product category of the product to be put on the shelf, and the similarity is comprehensively determined by the appearance picture and the text description information; receives the appearance compliance detection request message from the server The appearance compliance detection response message of the server includes at least one similar appearance design; and, at least one similar appearance design, the similarity score of each similar appearance design, and the feature comparison control corresponding to each similar appearance design are displayed on the product appearance compliance detection interface; a triggering operation of a feature comparison control of any reference similar appearance design in at least one similar appearance design is detected, and a feature comparison request message is created; and, a feature comparison request message is sent to the server, and the feature comparison request message is used to instruct the server to generate a feature comparison table of the product to be put on the shelf and the reference similar appearance design; a feature comparison response message is received from the server, and the feature comparison response message includes a feature comparison table; and, the feature comparison table is displayed on the product appearance compliance detection interface. Therefore, compared with the method of searching only using text descriptions in the prior art, the present application performs multimodal fusion retrieval through image-text fusion technology to obtain multiple similar appearance designs and feature comparison tables with the products to be put on the shelf, which significantly improves the effect of matching appearance patents with commodities and improves the comprehensiveness and accuracy of product appearance compliance detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0008] Figure 1 A schematic diagram of the structure of an e-commerce platform provided in an embodiment of the present application; Figure 2 A flowchart of a multi-modal fusion product appearance compliance detection method provided in an embodiment of the present application; Figure 3 A schematic diagram of the product appearance compliance test interface when a user uploads product image and text information provided in this application; Figure 4 The user provided in this application clicks the appearance inspection control to obtain a product appearance compliance inspection interface with multiple similar appearance designs; Figure 5 A schematic diagram of a product appearance compliance detection interface provided for this application, in which a user clicks on a feature comparison control of any similar appearance design to obtain a feature comparison table; Figure 6 A schematic diagram of the product appearance compliance testing interface after the product to be put on the shelf has been tested and found to be compliant, provided for this application; Figure 7 A schematic diagram of a product appearance compliance detection interface provided by the present application in which a user clicks on a design avoidance control to obtain at least one design avoidance appearance image; Figure 8 A schematic diagram of a product appearance compliance testing interface provided by this application in which a user clicks a test report creation control to obtain an appearance test report; Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application; Fig.10 A comparison table of product appearance features between the product to be put on the shelves (heater) and a reference similar appearance design (humidifier heater) provided in the embodiment of the present application. DETAILED DESCRIPTION

[0009] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0010] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0011] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0012] In the embodiments of the present application, "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist at the same time; B exists alone. Among them, A and B can be singular or plural.

[0013] In the embodiment of the present application, the symbol " / " can indicate that the objects associated with each other are in an "or" relationship. In addition, the symbol " / " can also indicate a division sign, that is, performing a division operation. For example, A / B can indicate A divided by B.

[0014] In the embodiments of the present application, "at least one item" or similar expressions refer to any combination of these items, including any combination of single items or plural items, and refer to one or more, and multiple refers to two or more. For example, at least one item of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.

[0015] In the embodiments of the present application, "equal to" can be used in conjunction with greater than, and is applicable to the technical solution adopted when greater than, and can also be used in conjunction with less than, and is applicable to the technical solution adopted when less than. When equal to is used in conjunction with greater than, it is not used in conjunction with less than; when equal to is used in conjunction with less than, it is not used in conjunction with greater than.

[0016] The present application provides a multimodal fusion product appearance compliance detection method and system, which performs multimodal fusion retrieval through image-text fusion technology to obtain multiple similar appearance designs and feature comparison tables with the products to be put on the shelves, significantly improving the effect of matching appearance patents with commodities, and improving the comprehensiveness and accuracy of product appearance compliance detection.

[0017] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0018] See also Figure 1 to Figure 2 , Figure 1A schematic diagram of the structure of an e-commerce platform provided in an embodiment of the present application, Figure 2 A flowchart of a multimodal fusion product appearance compliance detection method provided in an embodiment of the present application.

[0019] The e-commerce platform 1 includes a terminal device 10 and a server 20, and the terminal device 10 and the server 20 are connected to each other in a wired or wireless manner.

[0020] The terminal device 10 may specifically include a front-end device applied to the user side and capable of realizing functions such as data collection and data transmission, and may be a user equipment (UE) such as a mobile phone, a smart phone, a laptop, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing equipment connected to a wireless modem, a mobile station (MS), a mobile terminal, etc. Alternatively, the terminal device 10 may also be a software application that can be run in the above electronic devices. For example, it may be an APP running on a mobile phone.

[0021] The server 20 may specifically include a server for data processing in the background that is applied to one side of the network platform and can realize functions such as data transmission and data processing. It may be a physical server or a server cluster or distributed system composed of multiple physical servers. In this embodiment, the number of servers is not specifically limited. Alternatively, it may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0022] The terminal device 10 is Figure 2 The execution subject of a multi-modal fusion product appearance compliance detection method shown in the figure includes the following steps S201 to S204: Step S201, the terminal device obtains the appearance picture and text description information of the product to be put on the shelf uploaded by the user, and displays the appearance detection control on the product appearance compliance detection interface; and, detects the triggering operation of the appearance detection control, creates an appearance compliance detection request message; and sends the appearance compliance detection request message to the server.

[0023] Among them, after the user uploads the appearance picture and text description information of the product to be listed on the product appearance compliance detection interface, after completing the above operations, clicks the appearance detection control; the terminal device creates an appearance compliance detection request message based on the appearance picture and text description information, and sends the appearance compliance detection request message to the server.

[0024] The text description information includes but is not limited to the name of the design product, its purpose, key points of the product design (with emphasis on shape or pattern) and a description of the key design pictures.

[0025] Among them, the appearance compliance detection request message includes the appearance pictures and text description information of the product to be put on the shelf. The appearance compliance detection request message is used to instruct the server to filter out at least one similar appearance design from the appearance design database whose similarity with the product to be put on the shelf is greater than the preset similarity. The appearance design database includes the appearance pictures and text description information of multiple appearance designs corresponding to the product category of the product to be put on the shelf, and the similarity is comprehensively determined by the appearance pictures and text description information. That is, after receiving the appearance compliance detection request message from the terminal device, the server searches the appearance design database according to the appearance pictures and text description information of the product to be put on the shelf, and filters out at least one similar appearance design whose similarity with the product to be put on the shelf is greater than the preset similarity; the server feeds back at least one similar appearance design to the terminal device.

[0026] The design database may be a patent database, which includes appearance images and text description information of multiple patents, and the product categories of multiple patents are the same as the product categories of the products to be put on the shelves. In addition, the design data may also include general product images and text description information, and the product categories of general products are the same as the product categories of the products to be put on the shelves.

[0027] For specific implementation, please refer to Figure 3 Figure 3 A schematic diagram of the product appearance compliance detection interface when the user uploads product image and text information provided for this application, such as Figure 3 As shown, the product appearance compliance detection interface 3 includes a product display area 31. Clicking the newly added control in the product display area 31 can display the file upload area 32, which is used to upload the appearance picture and text description information of the product to be put on the shelf. Clicking other products such as product 1 can display the appearance detection results of product 1. After the user uploads the image and text information of the product to be put on the shelf, click the appearance detection control to perform appearance compliance detection.

[0028] Step S202, the terminal device receives an appearance compliance detection response message from the server and displays at least one similar appearance design, a similarity score of each similar appearance design, and a feature comparison control corresponding to each similar appearance design on the product appearance compliance detection interface.

[0029] Among them, the terminal device receives an appearance compliance detection response message from the server, which includes the at least one similar appearance design, and displays the at least one similar appearance design on the product appearance compliance detection interface. The displayed content includes the appearance picture and similarity of each similar appearance design, and a feature comparison control is set under each appearance picture.

[0030] When displaying at least one similar appearance design, the terminal device may sort and display the at least one similar appearance design according to the degree of similarity between the at least one similar appearance design and the product to be listed.

[0031] In a specific implementation, after the terminal device receives at least one similar appearance design, a similar appearance design area 33 is displayed on the product appearance compliance detection interface 3, and the similar appearance design area 33 includes at least one similar appearance design, such as Figure 4 As shown, each appearance design includes an appearance picture, similarity, and feature comparison controls.

[0032] Step S203: the terminal device detects a triggering operation on a feature comparison control of any reference similar design among the at least one similar design, creates a feature comparison request message; and sends the feature comparison request message to the server.

[0033] Among them, after the user clicks on the feature comparison control of any similar appearance design, the terminal device creates a feature comparison request message according to the trigger operation, and sends the feature comparison request message to the server, and the feature comparison request message is used to instruct the server to generate a feature comparison table between the product to be put on the shelf and the reference similar appearance design. That is, after receiving the feature comparison request message, the server generates a feature comparison table between the product to be put on the shelf and the reference similar appearance design, and sends the feature comparison table to the terminal device.

[0034] Among them, the feature comparison table is used to show the differences in appearance between the product to be listed and the reference similar design, such as Fig.10 shown.

[0035] Step S204, the terminal device receives a feature comparison response message from the server, and displays the feature comparison table on the product appearance compliance detection interface.

[0036] Among them, the terminal device receives a comparison response message from the server, the feature comparison response message includes the feature comparison table, and the feature comparison table is displayed on the product appearance compliance detection interface.

[0037] In specific implementation, after the user clicks a feature comparison control of a similar appearance design, a feature comparison table display area 34 is displayed on the product appearance compliance detection interface 3, such as Figure 5 As shown, the feature comparison table display area 34 displays Fig.10 The feature comparison table shown is used to display the feature comparison between the product to be listed and the reference similar appearance design.

[0038] In some embodiments, if the server fails to select an appearance design from the appearance design database whose similarity with the product to be listed is greater than a preset similarity, it indicates that there is no appearance design that is highly similar to the product to be listed in the current appearance design database, and the server sends a detection compliance prompt information to the terminal device; the terminal device displays the detection compliance prompt information on the product appearance compliance detection interface.

[0039] In specific implementation, when the server does not find a similar design with a similarity greater than a preset similarity to the product to be put on the shelf, the similar design area 33 displays a detection compliance prompt message, such as Figure 6 shown.

[0040] It can be seen that in the embodiment of the present application, the terminal device obtains the appearance picture and text description information of the product to be put on the shelf uploaded by the user, and displays the appearance detection control on the product appearance compliance detection interface; and, upon detecting the triggering operation of the appearance detection control, creates an appearance compliance detection request message, the appearance compliance detection request message includes the appearance picture and text description information; and, sends the appearance compliance detection request message to the server, the appearance compliance detection request message is used to instruct the server to filter out at least one similar appearance design whose similarity with the product to be put on the shelf is greater than a preset similarity from the appearance design database, the appearance design database includes the appearance pictures and text description information of multiple appearance designs corresponding to the product category of the product to be put on the shelf, and the similarity is comprehensively determined by the appearance picture and the text description information; receives the appearance compliance detection request message from the server The appearance compliance detection response message of the server includes at least one similar appearance design; and, at least one similar appearance design, the similarity score of each similar appearance design, and the feature comparison control corresponding to each similar appearance design are displayed on the product appearance compliance detection interface; a triggering operation of a feature comparison control of any reference similar appearance design in at least one similar appearance design is detected, and a feature comparison request message is created; and, a feature comparison request message is sent to the server, and the feature comparison request message is used to instruct the server to generate a feature comparison table of the product to be put on the shelf and the reference similar appearance design; a feature comparison response message is received from the server, and the feature comparison response message includes a feature comparison table; and, the feature comparison table is displayed on the product appearance compliance detection interface. Therefore, compared with the method of searching only using text descriptions in the prior art, the present application performs multimodal fusion retrieval through image-text fusion technology to obtain multiple similar appearance designs and feature comparison tables with the products to be put on the shelf, which significantly improves the effect of matching appearance patents with commodities and improves the comprehensiveness and accuracy of product appearance compliance detection.

[0041] In some embodiments, the process of the server screening the at least one similar design specifically includes the following steps a to d: Step a, the server obtains the appearance picture and text description information of the product to be put on the shelf, and obtains the appearance design vector database of the appearance design database, wherein the appearance design vector database includes the product appearance fusion feature vectors of the multiple appearance designs; Among them, the product appearance fusion feature vector refers to the comprehensive feature vector obtained after fusing the product's appearance picture and text description information, that is, the product appearance fusion feature vector covers the contents of both the appearance picture and the text description information.

[0042] Step b, the server vectorizes the appearance picture and text description information of the product to be put on the shelf to obtain a fusion feature vector of the target product appearance; Step c: the server performs a vector search on the design vector database according to the target product appearance fusion feature vector to obtain at least one product appearance fusion feature vector having a similarity with the target product appearance fusion feature vector greater than a preset similarity; In step d, the server determines the at least one similar appearance design according to the appearance design corresponding to the at least one product appearance fusion feature vector.

[0043] The specific process of step b "the server vectorizes the appearance picture and text description information of the product to be put on the shelf to obtain the target product appearance fusion feature vector" includes the following steps b1-b2: Step b1, the server obtains an image feature vector according to the appearance picture of the product to be put on the shelf, and obtains a text feature vector according to the text description information of the product to be put on the shelf; Step b2, the server determines the target product appearance fusion feature vector based on the image feature vector, the text feature vector and a first weight ratio relationship between the image feature vector and the text feature vector, wherein the first weight ratio relationship is used to characterize the importance of the appearance picture and text description information to the target product appearance fusion feature vector.

[0044] Wherein, step b1 "the server obtains an image feature vector according to the appearance picture of the product to be put on the shelf" includes the following steps b11-b13: Step b11, the server obtains shape features and pattern features of the appearance picture of the product to be put on the shelf, and obtains product dimensions of the product to be put on the shelf, where the product dimensions include flat products and three-dimensional products; Among them, edge detection algorithms (such as Canny edge detection) can be used to identify edges and contours in the image, thereby extracting the shape features of the product. Gray-level co-occurrence matrix (GLCM) is used to extract texture information in the image, thereby extracting the pattern features of the product.

[0045] Among them, according to the different dimensions of the products, the products are divided into two-dimensional products and three-dimensional products. For two-dimensional products, such as carpets, wallpapers, floral fabrics, etc., the design focus is on the pattern, and the shape is used as an auxiliary factor; for three-dimensional products, such as recorders, televisions, and transportation vehicles, the design focus is on the shape, and the pattern and color are used as auxiliary factors. The difference in color is generally not used alone as a condition for judging similarity.

[0046] Therefore, the method for determining the second weight proportion relationship includes the following steps: if the product dimension is the planar product, the first weight of the shape feature is less than the second weight of the pattern feature, and the sum of the first weight and the second weight is 1; if the product dimension is the three-dimensional product, the first weight is greater than the second weight.

[0047] In specific implementation, the appearance picture is divided into several regions ri, the shape features and pattern features of each region ri are obtained, the shape feature vector and the pattern feature vector are obtained according to the graphic features and the pattern features, the image feature vector of each region ri is determined according to the second weight proportion relationship and the shape feature vector and the pattern feature vector, and the image feature vector of the appearance picture is obtained by performing mean calculation on the image feature vectors of each region ri.

[0048] Step b12, the server determines a second weight ratio relationship between the shape feature and the pattern feature according to the product dimension, wherein the second weight ratio relationship is used to represent the importance of the shape feature and the pattern feature in different product dimensions; Step b13: The server obtains the image feature vector according to the second weight ratio, the shape feature, and the pattern feature.

[0049] It can be seen that in this embodiment, through the edge detection algorithm and texture analysis method, the server can accurately extract the shape features and pattern features of the product from the image, and assign different weights to the shape features and pattern features according to the different dimensions of the product, taking into account the design differences between flat products and three-dimensional products, thereby improving the pertinence and accuracy of the image feature vector.

[0050] Among them, for step b2, the server obtains an image feature vector Vimage according to the appearance picture, obtains at least one text feature vector Vtext1, Vtext2, ..., Vtextn according to the text description information, and performs the following steps b21-b24 according to the image feature vector and the text feature vector to calculate and obtain the target product appearance fusion feature vector: Step b21: The server calculates the dot product of the image feature vector and each text feature vector to obtain the attention score: score1=Vimage·Vtext1; score2=Vimage·Vtext2; … scoren=Vimage·Vtextn; In step b22, the server uses the softmax function to normalize the attention score and obtain the attention weight Wi: ; Step b23, the server applies the attention weight to each text feature vector to obtain a weighted text feature vector Vtext': ; Step b24, the server determines the target product appearance fusion feature vector Vtext_image=w1×Vtext'+w2×Vimage according to the weighted text feature vector Vtext', the image feature vector Vimage and the first weight ratio relationship (text feature vector: image feature vector=w1:w2, w1+w2=1).

[0051] Among them, the method for determining the first weight ratio relationship includes the following steps: obtaining all image features of the appearance picture, the image features include overall image features and local image features; determining text features according to the text description content; judging the overlap between the image features and the text features, if the overlap is high, it indicates that the text features cannot provide other valid information other than the image features, then the weight of the text features is reduced; if the overlap is low, it indicates that the text features can provide other valid information other than the image features, then the weight of the text features is appropriately increased; determine the first weight ratio relationship according to the weights of the text features and the image features.

[0052] For example, the text description of product 1 to be put on the shelves is red sneakers, and the system obtains the text features "red" and "sneakers". The system can obtain image features such as color, shape of sneakers, style of sneakers, and brand based on the appearance picture; since the image features include the content of text features, the weight of text features can be reduced; the text description of product 2 to be put on the shelves is red leather shoes, the material is crocodile leather, and the style is retro, and the system obtains text features "red", "leather shoes", "crocodile leather", and "retro"; the system can obtain image features such as color, shape of sneakers, style of sneakers, and brand based on the appearance picture; since the image features do not include the material, style and other features of the products to be put on the shelves, the weight of text features can be increased.

[0053] Specifically, the overlap between text features and image features can be determined by the similarity between text features and image features: obtain the image feature vector Vimage according to the image features, and obtain the text feature vector Vtext' according to the text features; calculate the cosine similarity Stext_image between the text feature vector Vtext' and the image feature vector Vimage: ; If the cosine similarity Stext_image is greater than the preset threshold, it is considered that the overlap is high, and the weight of the text feature w1, w1 = w1 × (1-δ) is reduced; if the cosine similarity Stext_image is less than the preset threshold, it is considered that the overlap is low, and the weight of the text feature w1, w1 = w1 × (1 + δ) is reduced, where δ is the adjustment factor, δ∈[0,1], which controls the range of weight change.

[0054] For example, if the initial weights of text features and image features are w1=0.5, w2=0.5, δ=0.1, if the overlap is high, w1=w1×(1-δ)=0.5×(1-0.1)=0.45, then w2=1-w1=0.55; if the overlap is low, w1=w1×(1+δ)=0.5×(1+0.1)=0.55, then w2=1-w1=0.45.

[0055] The appearance pictures and text description information of each appearance design in the appearance design database are processed as in steps b1 to b2 to obtain an appearance design vector database.

[0056] It can be seen that in this embodiment, the server obtains the product appearance fusion feature vector of the product to be put on the shelf through image-text fusion calculation based on the text description information and appearance picture of the product to be put on the shelf, so as to understand the characteristics of the product more accurately.

[0057] Among them, for step c "the server performs vector retrieval on the appearance design vector database according to the target product appearance fusion feature vector, and obtains at least one product appearance fusion feature vector whose similarity with the target product appearance fusion feature vector is greater than the preset similarity", it specifically includes: calculating the vector distance between the target product appearance fusion feature vector and each product appearance fusion feature vector in the appearance design vector database; determining the corresponding preset vector distance according to the preset familiarity, the vector distance and the similarity are inversely correlated, that is, the smaller the vector distance, the higher the similarity, and the larger the vector distance, the smaller the similarity; using the product appearance fusion feature vector whose vector distance is less than the preset vector distance as the aforementioned at least one product appearance fusion feature vector.

[0058] It can be seen that in this embodiment, the server obtains the product appearance fusion feature vector based on the product's appearance picture and text description information, and performs vector retrieval on the appearance design vector database based on the target product appearance fusion feature vector of the product to be listed, so as to more accurately find the appearance design similar to the target product.

[0059] In some embodiments, the process of the server creating the feature comparison table includes the following steps e-step I: Step e, the server obtains the image feature vector of the reference similar design and the image feature vector of the product to be put on the shelf; The server obtains the image feature vector Vtarget of the product to be put on the shelf and the image feature vector Vref of the reference similar appearance design selected by the user according to the aforementioned step b1, where Vtarget and Vref both include the image feature vectors Vtarget(ri), Vref(ri) of several regions ri of the appearance picture; Step f, the server determines at least one region where the product to be put on the shelf is different from the reference similar design and the image feature vector of the product to be put on the shelf, and a difference in the image feature vector of the at least one region according to the image feature vector of the reference similar design and the image feature vector of the product to be put on the shelf; Among them, the eigenvector difference of each region is calculated: △V(ri)=|Vtarget(ri)-Vref(ri)|.

[0060] Step g, the server determines a distinguishing feature comparison point from the at least one area according to a size relationship between the image feature vector difference and a preset difference; Among them, the preset difference value is ε, and the distinguishing feature point R={ri|△V(ri)≥ε} is determined according to the size relationship between the image feature vector difference △V(ri) and the preset difference ε.

[0061] Step h, the server generates text description content of the distinguishing feature comparison point according to the image feature vector difference of the distinguishing feature comparison point; Among them, the text description of the specific difference in shape or pattern between the product to be put on the shelf and the reference similar design at the distinguishing feature point ri∈R is determined according to the difference in the image feature vector of the distinguishing feature point ri∈R. For example, "In area r1, the edge of the product to be put on the shelf is smoother, while the edge of the reference design is sharper." In step i, the server creates the feature comparison table according to the distinguishing feature comparison points and their corresponding text description contents.

[0062] Among them, a feature comparison table is generated according to the text description content of each distinguishing feature point ri∈R to form a structured data display.

[0063] For example, see Fig.10 , Fig.10 A comparison table of product appearance features of the product to be put on the shelf (heater) provided in the embodiment of the present application and the reference similar appearance design (humidifier heater), composed of Fig.10It can be seen that the overall appearance of the reference similar appearance design and the product to be put on the shelf is relatively similar, but there are also differences. In the front view, rear view, and left view, the position relationship between the air distribution net tube and the heater body of the product to be put on the shelf is different from that of the reference similar appearance design, and has an impact on the front viewing effect; in the rear view, the position of the ventilation grid and groove of the product to be put on the shelf is different from that of the reference similar appearance design, which has an impact on the viewing effect at the back; in the top view, the top design of the product to be put on the shelf is different from the reference similar appearance design, and the product to be put on the shelf is a smooth plane, which has an impact on the viewing effect on the top.

[0064] It can be seen that in this embodiment, through image feature extraction and difference calculation, the areas where there are significant differences between the products to be put on the shelves and the reference design can be accurately identified, the image feature vector differences in the areas where there are significant differences are converted into text description content, and a feature comparison table is constructed based on the text description content, which improves the work efficiency of product appearance comparison and enables users to understand the specific differences between different products at a glance.

[0065] In some embodiments, after displaying the feature comparison table on the product appearance compliance detection interface, the method further includes: displaying a circumvention design control on the product appearance compliance detection interface; receiving a trigger operation on the circumvention design control, creating a circumvention design request message, the circumvention design request message carrying the feature comparison table; sending the circumvention design request message to the server, the circumvention design request message being used to instruct the server to generate a circumvention design appearance diagram according to the feature comparison table; receiving a circumvention design response message from the server, the circumvention design response message including the circumvention design appearance diagram; and displaying the circumvention design appearance diagram on the product appearance compliance detection interface.

[0066] Among them, after the terminal device displays the feature comparison table on the product appearance compliance detection interface, it displays the avoidance design control; the user clicks the avoidance design control, the terminal device creates a avoidance design request message, and sends the avoidance design request message to the server; after the server receives the avoidance design request message, it generates a avoidance design appearance diagram based on the feature comparison table, and feeds back the avoidance design appearance diagram to the terminal device; after the terminal device receives the avoidance design appearance diagram, it displays the avoidance design appearance diagram on the product appearance compliance detection interface.

[0067] Furthermore, the product compliance appearance inspection interface also displays a test report creation control. When the user clicks the test report creation control, the terminal device generates an appearance inspection report based on the avoidance design appearance diagram and the feature comparison table. Alternatively, the terminal device directly generates an appearance inspection report based on the feature comparison table, and the user can click to download the appearance inspection report.

[0068] Furthermore, the server can generate multiple circumvention design drawings for different distinguishing feature comparison points in the feature comparison table and different adjustment methods for different distinguishing feature comparison points. The user can select one or more appearance drawings from the multiple circumvention design appearance drawings to generate an appearance inspection report.

[0069] When implementing it specifically, Figure 5 As shown, after the terminal device displays the feature comparison table on the product appearance compliance detection interface 3, the circumvention design control and the test report creation control are also displayed on the product appearance compliance detection interface 3. When the user clicks the circumvention design control, the circumvention design appearance display area 35 is displayed on the product compliance detection interface 3, as shown in FIG. Figure 7 As shown, one or more circumvention design appearance drawings are displayed in the circumvention design appearance drawing display area 35, and Figure 7 The product compliance detection interface 3 shown in the figure displays a detection report creation control. After the user selects one or more circumvention design appearance drawings, he clicks the detection report creation control. The terminal device generates an appearance detection report according to the selected one or more circumvention design appearance drawings and the feature comparison table, such as Figure 8 When the user clicks Figure 5 After the test report shown in the figure is created, the control is displayed as follows Figure 8 The interface shown.

[0070] It can be seen that in this embodiment, by triggering the design avoidance control, the user can instantly create and send a design avoidance request message. The server quickly processes and returns the design avoidance appearance image without switching to other platforms or tools. This improves the compliance detection function of the e-commerce platform and enhances the user experience.

[0071] In some embodiments, the process of the server creating the circumvention design appearance image includes the following steps j-k: Step j, the server obtains the feature comparison table and the image feature vector of the product to be put on the shelf; Step k, the server generates one or more appearance avoidance design text solutions according to the distinguishing feature comparison points in the feature comparison table and the image feature vectors of the product to be put on the shelf; The process of the server creating the one or more appearance avoidance design text solutions specifically includes the following steps k1 to k3: Step k1, the server obtains the feature importance of the distinguishing feature comparison point in the feature comparison representation in the appearance design of the product to be put on the shelf; Among them, the feature importance of the distinguishing feature comparison points can be determined based on the general attention range of general consumers to the products on the shelves and the degree of influence of the distinguishing feature comparison points on the overall appearance. Specifically, for the distinguishing feature comparison points that consumers can clearly perceive, their feature importance is higher. The greater the difference in appearance compared to the reference similar appearance design, the less likely it is to be identified as the same or similar appearance; for the distinguishing feature comparison points that consumers cannot easily perceive, their feature importance is lower, and no special requirements are required for the difference in appearance compared to the reference similar appearance design. Regarding the degree of influence of the distinguishing feature comparison points on the overall appearance, the degree of influence can be determined by factors such as the relative position, size, shape, and color of the distinguishing feature comparison points in the overall appearance. The feature importance is determined comprehensively through the general attention range and the degree of influence.

[0072] Step k2, the server determines an adjustment amplitude of the image feature vector of the distinguishing feature comparison point according to the feature importance, and the adjustment amplitude is positively correlated with the feature importance; Step k3: the server generates one or more appearance avoidance design text solutions according to the adjustment range and the image feature vector of the product to be put on the shelf.

[0073] Step 1: The server generates the appearance circumvention design diagram according to the one or more appearance circumvention design text solutions.

[0074] Among them, the server can use a graphic generation model such as BigGAN to generate an appearance avoidance design diagram.

[0075] It can be seen that in this embodiment, the server ensures that key features are adjusted preferentially by evaluating the importance of the features, thereby more effectively avoiding design similarity.

[0076] It should be noted that the specific implementation process of this embodiment can refer to the specific implementation process described in the above method embodiment, which will not be described here.

[0077] With the above Figure 2 For details on the embodiments shown in the drawings, please refer to Fig. 9 , Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application, such as Fig. 9 As shown, the terminal device 10 includes a processor 91, a memory 93, a communication interface 92, and one or more programs 931. The one or more programs 931 are stored in the memory 93 and are configured to be executed by the processor 91. The above-mentioned programs include methods for executing the methods described in the above-mentioned embodiments.

[0078] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0079] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0082] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0085] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0086] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A product appearance compliance detection method based on multimodal fusion, characterized in that: The method is applied to a terminal device of an e-commerce platform, and the e-commerce platform also includes a server. The method includes: Obtaining the appearance picture and text description information of the product to be put on the shelf uploaded by the user, and displaying the appearance detection control on the product appearance compliance detection interface; and, detecting a triggering operation on the appearance detection control, creating an appearance compliance detection request message, the appearance compliance detection request message including the appearance picture and the text description information; and, Sending the appearance compliance detection request message to the server, wherein the appearance compliance detection request message is used to instruct the server to filter out at least one similar appearance design from an appearance design database whose similarity to the product to be put on the shelf is greater than a preset similarity, wherein the appearance design database includes appearance pictures and text description information of multiple appearance designs corresponding to the product category of the product to be put on the shelf, and the similarity is comprehensively determined by the appearance pictures and text description information; receiving an appearance compliance detection response message from the server, the appearance compliance detection response message including the at least one similar appearance design; and displaying the at least one similar appearance design, a similarity score of each similar appearance design, and a feature comparison control corresponding to each similar appearance design on the product appearance compliance detection interface; Detecting a triggering operation of a feature comparison control for any reference similar design among the at least one similar design, creating a feature comparison request message; and sending the feature comparison request message to the server, wherein the feature comparison request message is used to instruct the server to generate a feature comparison table between the product to be put on the shelf and the reference similar design; receiving a feature comparison response message from the server, wherein the feature comparison response message includes the feature comparison table; and displaying the feature comparison table on the product appearance compliance detection interface.

2. The method according to claim 1, characterized in that The process of determining at least one similar design specifically comprises the following steps: Obtaining appearance pictures and text description information of the product to be put on the shelves, and obtaining a design vector database of the design database, wherein the design vector database includes product appearance fusion feature vectors of the multiple designs; Vectorize the appearance pictures and text description information of the product to be put on the shelf to obtain a fusion feature vector of the target product appearance; Performing a vector search on the appearance design vector database according to the target product appearance fusion feature vector to obtain at least one product appearance fusion feature vector having a similarity with the target product appearance fusion feature vector greater than a preset similarity; The at least one similar appearance design is determined according to the appearance design corresponding to the at least one product appearance fusion feature vector.

3. The method according to claim 2, characterized in that The step of vectorizing the appearance picture and text description information of the product to be put on the shelf to obtain a target product appearance fusion feature vector includes: Obtaining an image feature vector based on the appearance picture of the product to be put on the shelf, and obtaining a text feature vector based on the text description information of the product to be put on the shelf; The target product appearance fusion feature vector is determined based on the image feature vector, the text feature vector, and a first weight ratio relationship between the image feature vector and the text feature vector. The first weight ratio relationship is used to characterize the importance of the appearance picture and text description information to the target product appearance fusion feature vector.

4. The method according to claim 3, characterized in that The image feature vector is obtained according to the appearance picture of the product to be put on the shelf, including: Obtaining shape features and pattern features of the appearance picture of the product to be put on the shelf, and obtaining product dimensions of the product to be put on the shelf, wherein the product dimensions include flat products and three-dimensional products; Determine a second weight ratio relationship between the shape feature and the pattern feature according to the product dimension, wherein the second weight ratio relationship is used to characterize the importance of the shape feature and the pattern feature in different product dimensions; The image feature vector is obtained according to the second weight proportion relationship, the shape feature, and the pattern feature.

5. The method according to claim 4, characterized in that The method for determining the second weight proportion relationship comprises the following steps: If the product dimension is the flat product, the first weight of the shape feature is less than the second weight of the pattern feature, and the sum of the first weight and the second weight is 1; If the product dimension is the three-dimensional product, the first weight is greater than the second weight.

6. The method according to claim 4, characterized in that The process of creating the feature comparison table includes the following steps: Obtaining an image feature vector of the reference similar design and an image feature vector of the product to be put on the shelf; Determine at least one region where the product to be put on the shelf is different from the reference similar design and the image feature vector of the product to be put on the shelf, and the image feature vector difference of the at least one region according to the image feature vector of the reference similar design and the image feature vector of the product to be put on the shelf; Determining a distinguishing feature comparison point from the at least one area according to a magnitude relationship between the image feature vector difference and a preset difference; Generating text description content of the distinguishing feature comparison point according to the image feature vector difference of the distinguishing feature comparison point; The feature comparison table is created according to the distinguishing feature comparison points and their corresponding text description contents.

7. The method according to claim 6, characterized in that After the feature comparison table is displayed on the product appearance compliance detection interface, the method further includes: Displaying a design avoidance control on the product appearance compliance detection interface; receiving a triggering operation on the design circumvention control, and creating a design circumvention request message, wherein the design circumvention request message carries the feature comparison table; Sending the design circumvention request message to the server, wherein the design circumvention request message is used to instruct the server to generate a design circumvention appearance diagram according to the feature comparison table; receiving a design circumvention response message from the server, wherein the design circumvention response message includes the design circumvention appearance diagram; The circumvention design appearance diagram is displayed on the product appearance compliance detection interface.

8. The method according to claim 7, characterized in that The process of creating the circumvention design appearance diagram includes the following steps: Obtaining the feature comparison table, and obtaining the image feature vector of the product to be put on the shelf; One or more appearance circumvention design text solutions are generated according to the distinguishing feature comparison points in the feature comparison table and the image feature vectors of the products to be put on the shelves; and the appearance circumvention design drawings are generated according to the one or more appearance circumvention design text solutions.

9. The method according to claim 8, characterized in that The process of creating one or more appearance avoidance design text solutions specifically includes the following steps: Obtaining the feature importance of the distinguishing feature comparison point in the feature comparison representation in the appearance design of the product to be put on the shelf; Determining an adjustment amplitude of the image feature vector of the distinguishing feature contrast point according to the feature importance, wherein the adjustment amplitude is positively correlated with the feature importance; The one or more appearance avoidance design text solutions are generated according to the adjustment range and the image feature vector of the product to be put on the shelf.

10. An e-commerce platform system, characterized in that: The e-commerce platform system includes a terminal device and a server, and the terminal device is used to execute the step instructions in the method as described in any one of claims 1-9.

Images