Product appearance compliance detection method and system based on multi-modal fusion

Abstract

The application provides a multi-modal fusion product appearance compliance detection method and system. A terminal device acquires appearance pictures and text description information of a product to be put on the market uploaded by a user, and sends an appearance compliance detection request message to a server after detecting a trigger operation on an appearance detection control. At least one similar appearance design is received from the server. The at least one similar appearance design and corresponding similarity scores and feature comparison controls are displayed on a product appearance compliance detection interface. A feature comparison request message is sent to the server after detecting a trigger operation on any feature comparison control. A feature comparison table is received from the server. The feature comparison table is displayed on the product appearance compliance detection interface. Therefore, the application performs multi-modal fusion retrieval through image-text fusion technology to obtain multiple similar appearance designs and a feature comparison table with the product to be put on the market, thereby improving the comprehensiveness and accuracy of product appearance compliance detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and in particular to a multi-modal fusion product appearance compliance detection method and system. BACKGROUND

[0002] In the compliance scenario, the compliance detection of the appearance patent and the product to be listed is a key task to ensure the appearance compliance of the product to be listed. The traditional matching method mainly relies on the text description, but since most of the information in the appearance patent is presented in the form of images, the text description is often short, which limits the matching effect. The application of natural language processing (NLP) technology in such scenarios is limited. SUMMARY

[0003] The present application provides a multi-modal fusion product appearance compliance detection method and system, which performs multi-modal fusion retrieval through a graphic-text fusion technology to obtain a plurality of similar appearance designs and a feature comparison table with the product to be listed, significantly improving the matching effect of the appearance patent and the commodity, and improving the comprehensiveness and accuracy of the product appearance compliance detection.

[0004] In a first aspect, the present application provides a multi-modal fusion product appearance compliance detection method, which is applied to a terminal device of an e-commerce platform, the e-commerce platform further comprising a server, the method comprising: obtaining an appearance picture and text description information of a product to be listed uploaded by a user, and displaying an appearance detection control on a product appearance compliance detection interface; and detecting a triggering operation on the appearance detection control, creating an appearance compliance detection request message, the appearance compliance detection request message comprising the appearance picture and the text description information; and sending the appearance compliance detection request message to the server, the appearance compliance detection request message being used to instruct the server to filter at least one similar appearance design with a similarity greater than a preset similarity to the product to be listed from an appearance design database, the appearance design database comprising appearance pictures and text description information of a plurality of appearance designs corresponding to a product category of the product to be listed, the similarity being determined by the appearance picture and the text description information;

[0005] receiving an appearance compliance detection response message from the server, the appearance compliance detection response message including the at least one similar appearance design; and displaying, on the product appearance compliance detection interface, the at least one similar appearance design, and a similarity score of each similar appearance design, and a feature comparison control corresponding to each similar appearance design; detecting a triggering operation on the feature comparison control of any one reference similar appearance design in the at least one similar appearance design, creating a feature comparison request message; and sending the feature comparison request message to the server, the feature comparison request message being used to instruct the server to generate a feature comparison table of the product to be listed and the reference similar appearance design; receiving a feature comparison response message from the server, the feature comparison response message including the feature comparison table; and displaying, on the product appearance compliance detection interface, the feature comparison table.

[0006] In a second aspect, the present application provides an e-commerce platform system, which comprises a terminal device and a server. The terminal device is configured to execute the step instructions in the method of any one of the first aspect.

[0007] It can be seen that in the embodiments of the present application, the terminal device acquires the appearance picture and the text description information of the product to be uploaded uploaded by the user, displays the appearance detection control on the product appearance compliance detection interface, detects the triggering operation of the appearance detection control, creates an appearance compliance detection request message, the appearance compliance detection request message includes the appearance picture and the text description information, and sends the appearance compliance detection request message to the server. The appearance compliance detection request message is used to instruct the server to filter at least one similar appearance design with a similarity greater than a preset similarity from the appearance design database corresponding to the product category of the product to be uploaded. The appearance design database includes the appearance pictures and the text description information of a plurality of appearance designs corresponding to the product category of the product to be uploaded. The similarity is determined by the appearance picture and the text description information. The appearance compliance detection response message from the server is received, the appearance compliance detection response message includes at least one similar appearance design, and the at least one similar appearance design, the similarity score of each similar appearance design, and the feature comparison control corresponding to each similar appearance design are displayed on the product appearance compliance detection interface. The triggering operation of the feature comparison control of any reference similar appearance design in the at least one similar appearance design is detected, a feature comparison request message is created, and the feature comparison request message is sent to the server. The feature comparison request message is used to instruct the server to generate a feature comparison table of the product to be uploaded and the reference similar appearance design. The feature comparison response message from the server is received, the feature comparison response message includes the feature comparison table, and the feature comparison table is displayed on the product appearance compliance detection interface. Therefore, compared with the method of using only text description for retrieval in the prior art, the present application performs multi-modal fusion retrieval through image-text fusion technology to obtain a plurality of similar appearance designs and a feature comparison table with the product to be uploaded, which significantly improves the effect of matching the appearance patent with the commodity and improves the comprehensiveness and accuracy of the product appearance compliance detection. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0009] Figure 1 A structural schematic diagram of an e-commerce platform provided by an embodiment of the present application;

[0010] Figure 2 A flowchart of a multi-modal fusion product appearance compliance detection method provided by an embodiment of the present application;

[0011] Figure 3A schematic diagram of a product appearance compliance detection interface provided by the present application when a user uploads product image information;

[0012] Figure 4 A schematic diagram of a product appearance compliance detection interface provided by the present application when a user clicks an appearance detection control to obtain a plurality of similar appearance designs;

[0013] Figure 5 A schematic diagram of a product appearance compliance detection interface provided by the present application when a user clicks a feature comparison control of any similar appearance design to obtain a feature comparison table;

[0014] Figure 6 A schematic diagram of a product appearance compliance detection interface provided by the present application after a product to be listed is detected for compliance;

[0015] Figure 7 A schematic diagram of a product appearance compliance detection interface provided by the present application when a user clicks an avoidance design control to obtain at least one avoidance design appearance diagram;

[0016] Figure 8 A schematic diagram of a product appearance compliance detection interface provided by the present application when a user clicks a detection report creation control to obtain an appearance detection report;

[0017] Figure 9 A schematic diagram of a structure of a terminal device provided by an embodiment of the present application;

[0018] Figure 10 A product appearance feature comparison table of a product to be listed (a warmer) and a reference similar appearance design (a humidifying warmer) provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor fall within the scope of protection of the present application.

[0020] The terms “first”, “second”, and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product, or device.

[0021] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that any of the embodiments described herein can be incorporated into any other embodiment in a manner known to those of ordinary skill in the art.

[0022] The“and / or” in the embodiments of the application describes the association relationship of the associated objects, which means that there can be three relationships. For example, A and / or B can represent the following three cases: A exists alone; A and B exist simultaneously; B exists alone. Wherein, A and B can be singular or plural.

[0023] In the embodiments of the application, the symbol“ / ” can represent that the associated objects before and after the symbol are in an“or” relationship. In addition, the symbol“ / ” can also represent the division sign, that is, performing division operation. For example, A / B can represent A divided by B.

[0024] The“at least one” or similar expressions in the embodiments of the application mean any combination of the items, including any combination of single item or multiple items, means one or more, and multiple means two or more. For example, at least one of a, b or c can represent the following seven cases: a, b, c, a and b, a and c, b and c, a, b and c. Wherein, each of a, b and c can be an element or a set containing one or more elements.

[0025] The“equal to” in the embodiments of the application can be combined with greater than, which is applicable to the technical solutions adopted when greater than, or combined with less than, which is applicable to the technical solutions adopted when less than. When equal to is combined with greater than, it is not combined with less than; when equal to is combined with less than, it is not combined with greater than.

[0026] The application provides a multi-modal fusion product appearance compliance detection method and system, which performs multi-modal fusion retrieval through graphic-text fusion technology, obtains multiple similar appearance designs and a feature comparison table with a product to be listed, significantly improves the matching effect of appearance patents and commodities, and improves the comprehensiveness and accuracy of product appearance compliance detection.

[0027] The technical solutions of the application and how the technical solutions of the application solve the above technical problems will be described in detail in specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0028] Please refer toFigures 1 to 2 , Figure 1 A structural schematic diagram of an e-commerce platform provided by an embodiment of the present application, Figure 2 A flowchart of a multi-modal fusion product appearance compliance detection method provided by an embodiment of the present application.

[0029] The e-commerce platform 1 includes a terminal device 10 and a server 20, and the terminal device 10 and the server 20 are connected in communication through wired or wireless means.

[0030] The terminal device 10 specifically can include a front-end device applied to a user side and capable of realizing data acquisition, data transmission and the like, and can be a user equipment (User Equipment, UE) such as a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, a mobile station (Mobile Station, MS), a mobile terminal (Mobile Terminal) and the like. Alternatively, the terminal device 10 can also be a software application capable of running in the above electronic devices. For example, it can be an APP running on a mobile phone and the like.

[0031] The server 20 specifically can include a server applied to a network platform side and capable of realizing data transmission, data processing and the like, and responsible for data processing in the background, and can be a physical server, a server cluster or a distributed system composed of multiple physical servers, and the number of servers is not specifically limited in the embodiment. Alternatively, it can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0032] The terminal device 10 is an execution subject of a multi-modal fusion product appearance compliance detection method as shown in the figure, and the multi-modal fusion product appearance compliance detection method includes the following steps S201-S204: Figure 2 Step S201, the terminal device acquires appearance pictures and text description information of a product to be uploaded uploaded by a user, and displays an appearance detection control on a product appearance compliance detection interface; and detects a triggering operation on the appearance detection control, creates an appearance compliance detection request message, and sends the appearance compliance detection request message to the server.

[0033]

[0034] ​Wherein, the user uploads the appearance picture and the text description information of the product to be listed on the product appearance compliance detection interface, and after completing the above operation, clicks the appearance detection control; the terminal device creates an appearance compliance detection request message according to the appearance picture and the text description information, and sends the appearance compliance detection request message to the server.

[0035] Wherein, the text description information includes but is not limited to the name, purpose, product appearance design points (focus on shape or pattern) and design point picture description of the appearance design product.

[0036] Wherein, the appearance compliance detection request message includes the appearance picture and the text description information of the product to be listed, and the appearance compliance detection request message is used to instruct the server to filter at least one similar appearance design with a similarity greater than a preset similarity from the appearance design database corresponding to the product category of the product to be listed. The appearance design database includes the appearance picture and the text description information of a plurality of appearance designs corresponding to the product category of the product to be listed, and the similarity is determined by the appearance picture and the text description information. That is, after the server receives the appearance compliance detection request message from the terminal device, the appearance design database is searched according to the appearance picture and the text description information of the product to be listed, and at least one similar appearance design with a similarity greater than a preset similarity is filtered; the server feeds back at least one similar appearance design to the terminal device.

[0037] Wherein, the appearance design database can be an appearance patent database, and the appearance patent database includes the appearance picture and the text description information of a plurality of appearance patents, and the product categories of the plurality of appearance patents are the same as the product category of the product to be listed. In addition, the appearance design data can also include general product pictures and their text description information, and the product categories of the general products are the same as the product category of the product to be listed.

[0038] For specific implementation, please refer to Figure 3 Figure 3 The schematic diagram of the product appearance compliance detection interface provided by the present application when the user uploads the product picture and text information is as shown in Figure 3 The product appearance compliance detection interface 3 includes a product display area 31, clicking the add control of the product display area 31 can display a file upload area 32, and the file upload area 32 is used to upload the appearance picture and the text description information of the product to be listed. Clicking other products such as product 1 can display the appearance detection result of product 1. When the user uploads the picture and text information of the product to be listed, click the appearance detection control to perform appearance compliance detection.

[0039] Step S202, the terminal device receives the appearance compliance detection response message from the server, and displays the at least one similar appearance design, the similarity score of each similar appearance design, and the feature comparison control corresponding to each similar appearance design on the product appearance compliance detection interface.

[0040] The terminal device receives the appearance compliance detection response message from the server, wherein the appearance compliance detection response message includes the at least one similar appearance design, and the at least one similar appearance design is displayed on the product appearance compliance detection interface, and the display content includes the appearance picture of each similar appearance design, the similarity, and the feature comparison control arranged below each appearance picture.

[0041] The terminal device can display the at least one similar appearance design in a sorted manner according to the similarity between the at least one similar appearance design and the product to be listed.

[0042] In a specific implementation, when the terminal device receives the at least one similar appearance design, the terminal device displays a similar appearance design area 33 on the product appearance compliance detection interface 3, and the similar appearance design area 33 includes the at least one similar appearance design. Figure 4 As shown in FIG. 3, each piece of appearance design includes an appearance picture, a similarity, and a feature comparison control.

[0043] Step S203, the terminal device detects a triggering operation on the feature comparison control of any reference similar appearance design in the at least one similar appearance design, creates a feature comparison request message, and sends the feature comparison request message to the server.

[0044] When the user clicks the feature comparison control of any similar appearance design, the terminal device creates a feature comparison request message according to the triggering operation, and sends the feature comparison request message to the server. The feature comparison request message is used to instruct the server to generate a feature comparison table of the product to be listed and the reference similar appearance design. That is, the server generates the feature comparison table of the product to be listed and the reference similar appearance design after receiving the feature comparison request message, and sends the feature comparison table to the terminal device.

[0045] The feature comparison table is used to show the details of the difference in appearance between the product to be listed and the reference similar appearance design. Figure 10 As shown in FIG. 4.

[0046] Step S204, the terminal device receives the feature comparison response message from the server, and displays the feature comparison table on the product appearance compliance detection interface.

[0047] The terminal device receives a comparison response message from the server, the feature comparison response message includes the feature comparison table, and the feature comparison table is displayed on the product appearance compliance detection interface.

[0048] In a specific implementation, after a user clicks a feature comparison control of a similar appearance design, a feature comparison table display area 34 is displayed on the product appearance compliance detection interface 3, as shown in Figure 5 A feature comparison table is displayed in the feature comparison table display area 34, as shown in Figure 10 The feature comparison table is used to show the feature comparison between the product to be listed and the reference similar appearance design.

[0049] In some embodiments, if the server fails to select, from the appearance design database, an appearance design that is similar to the product to be listed and has a similarity greater than a preset similarity, it indicates that there is no appearance design that is highly similar to the product to be listed in the current appearance design database. In this case, the server sends a compliance detection prompt information to the terminal device, and the terminal device displays the compliance detection prompt information on the product appearance compliance detection interface.

[0050] In a specific implementation, when the server fails to query a similar appearance design that is similar to the product to be listed and has a similarity greater than a preset similarity, the similar appearance design area 33 displays the compliance detection prompt information, as shown in Figure 6

[0051] ​As can be seen, in this embodiment, the terminal device obtains the appearance image and text description information of the product to be listed uploaded by the user, displays the appearance inspection control on the product appearance compliance inspection interface; and, upon detecting a trigger operation on the appearance inspection control, creates an appearance compliance inspection request message, which includes the appearance image and text description information; and sends the appearance compliance inspection request message to the server, which instructs the server to filter at least one similar appearance design from the appearance design database that has a similarity greater than a preset similarity to the product to be listed. The appearance design database includes appearance images and text description information of multiple appearance designs corresponding to the product category of the product to be listed, and the similarity is determined by a combination of the appearance image and text description information; and receives a request from the server... The system executes a multimodal fusion retrieval method using image-text fusion technology. This method, compared to existing methods that rely solely on textual descriptions for retrieval, achieves this by using image-text fusion technology to obtain multiple similar designs and a feature comparison table with the product to be listed. This significantly improves the matching effect between design patents and products, enhancing the comprehensiveness and accuracy of product appearance compliance testing. Furthermore, it displays at least one similar design, along with a similarity score and a corresponding feature comparison control on the product appearance compliance testing interface. Upon detecting a trigger operation on the feature comparison control of any of the at least one similar design, a feature comparison request message is created. The system then sends the feature comparison request message to the server, instructing the server to generate a feature comparison table between the product to be listed and the reference similar designs. Finally, it receives a feature comparison response message from the server, including the feature comparison table, and displays the feature comparison table on the product appearance compliance testing interface.

[0052] In some embodiments, the process of the server screening the at least one similar appearance design specifically includes the following steps a-d:

[0053] Step a, the server obtains the appearance image and text description information of the product to be listed, as well as the appearance design vector database of the appearance design database, the appearance design vector database including the product appearance fusion feature vectors of the multiple appearance designs;

[0054] Among them, the product appearance fusion feature vector refers to the comprehensive feature vector obtained after fusing the product's appearance image and text description information. In other words, the product appearance fusion feature vector simultaneously covers the content of the appearance image and the text description information.

[0055] Step b: The server vectorizes the appearance image and text description information of the product to be listed to obtain the target product appearance fusion feature vector;

[0056] Step c: The server performs vector retrieval on the appearance design vector database based on the target product appearance fusion feature vector to obtain at least one product appearance fusion feature vector that has a similarity greater than the preset similarity with the target product appearance fusion feature vector.

[0057] Step d: The server determines the at least one similar appearance design based on the appearance design corresponding to the at least one product appearance fusion feature vector.

[0058] Specifically, the process of step b, "the server vectorizes the appearance image and text description information of the product to be listed to obtain the target product appearance fusion feature vector," includes the following steps b1-b2:

[0059] Step b1: The server obtains an image feature vector based on the appearance image of the product to be listed, and obtains a text feature vector based on the text description information of the product to be listed.

[0060] Step b2: The server determines the target product appearance fusion feature vector based on the image feature vector, the text feature vector, and the first weight ratio relationship between the image feature vector and the text feature vector. The first weight ratio relationship is used to characterize the importance of the appearance image and text description information to the target product appearance fusion feature vector.

[0061] Specifically, step b1, "The server obtains the image feature vector based on the appearance image of the product to be listed," includes the following steps b11-b13:

[0062] Step b11: The server obtains the shape features and pattern features of the appearance image of the product to be listed, as well as the product dimensions of the product to be listed, including two-dimensional products and three-dimensional products.

[0063] Edge detection algorithms (such as Canny edge detection) can be used to identify edges and contours in the image, thereby extracting the shape features of the product. Gray-level co-occurrence matrix (GLCM) is used to extract texture information from the image, thereby extracting the pattern features of the product.

[0064] Based on different product dimensions, products are divided into two-dimensional products and three-dimensional products. For two-dimensional products, such as carpets, wallpaper, and printed fabrics, the design focus is on the pattern, with shape as a secondary factor. For three-dimensional products, such as tape recorders, televisions, and transportation vehicles, the design focus is on the shape, with pattern and color as secondary factors. Color differences are generally not used alone as a criterion for judging similarity.

[0065] Therefore, the method for determining the second weight ratio includes the following steps: if the product dimension is the planar product, then the first weight of the shape feature is less than the second weight of the pattern feature, and the sum of the first weight and the second weight is 1; if the product dimension is the three-dimensional product, then the first weight is greater than the second weight.

[0066] In specific implementation, the appearance image is divided into several regions ri, and the shape features and pattern features of each region ri are obtained. Based on the graphic features and pattern features, shape feature vectors and pattern feature vectors are obtained. Based on the second weight ratio relationship and the shape feature vectors and pattern feature vectors, the image feature vector of each region ri is determined. The image feature vector of the appearance image is obtained by averaging the image feature vectors of each region ri.

[0067] Step b12: The server determines a second weight ratio relationship between the shape feature and the pattern feature based on the product dimension. The second weight ratio relationship is used to characterize the importance of the shape feature and the pattern feature in different product dimensions.

[0068] Step b13: The server obtains the image feature vector based on the second weight ratio relationship, the shape feature, and the pattern feature.

[0069] As can be seen, in this embodiment, by using edge detection algorithms and texture analysis methods, the server can accurately extract the shape and pattern features of the product from the image. Based on the different dimensions of the product, different weights are assigned to the shape and pattern features, taking into account the design differences between planar and three-dimensional products, thereby improving the targeting and accuracy of the image feature vector.

[0070] Specifically, for step b2, the server obtains an image feature vector Vimage based on the appearance image and at least one text feature vector Vtext1, Vtext2, ..., Vtextn based on the text description information. Then, based on the image feature vector and the text feature vector, steps b21-b24 are performed to calculate the target product appearance fusion feature vector:

[0071] Step b21: The server calculates the dot product of the image feature vector and each text feature vector to obtain the attention score.

[0072] score1 = Vimage·Vtext1;

[0073] score2 = Vimage·Vtext2;

[0074] …

[0075] scoren = Vimage·Vtextn;

[0076] In step b22, the server uses the softmax function to normalize the attention scores, obtaining the attention weights Wi:

[0077] ;

[0078] In step b23, the server applies attention weights to each text feature vector, resulting in a weighted text feature vector Vtext':

[0079] ;

[0080] In step b24, the server determines the target product appearance fusion feature vector Vtext_image = w1×Vtext'+w2×Vimage based on the weighted text feature vector Vtext', image feature vector Vimage, and the first weight ratio relationship (text feature vector: image feature vector = w1:w2, w1+w2=1).

[0081] The determination of the first weight ratio includes the following steps: obtaining all image features of the appearance image, including overall image features and local image features; determining text features based on the text description; judging the overlap between image features and text features; if the overlap is high, it indicates that the text features cannot provide other effective information besides image features, so the weight of the text features is reduced; if the overlap is low, it indicates that the text features can provide other effective information besides image features, so the weight of the text features is appropriately increased; and determining the first weight ratio based on the weights of text features and image features.

[0082] For example, the text description of product 1 to be listed is "red sneakers." The system obtains the text features "red" and "sneakers." Based on the appearance image, the system can obtain image features such as color, sneaker shape, sneaker style, and brand. Since image features include the content of text features, the weight of text features can be reduced. The text description of product 2 to be listed is "red leather shoes," made of crocodile skin, with a retro style. The system obtains the text features "red," "leather shoes," "crocodile skin," and "retro style." Based on the appearance image, the system can obtain image features such as color, sneaker shape, sneaker style, and brand. Since image features do not include the material and style features of the product to be listed, the weight of text features can be increased.

[0083] Specifically, the overlap between text features and image features can be determined by the similarity between the text features and image features: obtain the image feature vector Vimage based on the image features, and obtain the text feature vector Vtext' based on the text features; calculate the cosine similarity Stext_image between the text feature vector Vtext' and the image feature vector Vimage:

[0084] ;

[0085] If the cosine similarity Stext_image is greater than the preset threshold, the overlap is considered high, and the weight of the text feature w1 is reduced, w1=w1×(1-δ); if the cosine similarity Stext_image is less than the preset threshold, the overlap is considered low, and the weight of the text feature w1 is reduced, w1=w1×(1+δ), where δ is an adjustment factor, δ∈[0,1], which controls the magnitude of weight change.

[0086] For example, if the initial weights of text features and image features are w1=0.5, w2=0.5, and δ=0.1, if the overlap is high, w1=w1×(1-δ)=0.5×(1-0.1)=0.45, then w2=1-w1=0.55; if the overlap is low, w1=w1×(1+δ)=0.5×(1+0.1)=0.55, then w2=1-w1=0.45.

[0087] Specifically, the appearance images and text descriptions of each appearance design in the appearance design database are processed using steps b1-b2 as described above to obtain the appearance design vector database.

[0088] As can be seen, in this implementation, the server calculates the product appearance fusion feature vector of the product to be listed based on the text description information and appearance image of the product to be listed, so as to understand the characteristics of the product more accurately.

[0089] Specifically, step c, "The server performs vector retrieval on the appearance design vector database based on the target product appearance fusion feature vector to obtain at least one product appearance fusion feature vector whose similarity to the target product appearance fusion feature vector is greater than the preset similarity," includes: calculating the vector distance between the target product appearance fusion feature vector and each product appearance fusion feature vector in the appearance design vector database; determining the corresponding preset vector distance based on the preset similarity, where vector distance and similarity are inversely correlated, i.e., the smaller the vector distance, the higher the similarity, and the larger the vector distance, the lower the similarity; and selecting the product appearance fusion feature vector whose vector distance is less than the preset vector distance as the aforementioned at least one product appearance fusion feature vector.

[0090] As can be seen, in this embodiment, the server obtains the product appearance fusion feature vector based on the product's appearance image and text description information, and performs vector retrieval on the appearance design vector database based on the target product's appearance fusion feature vector, which can more accurately find appearance designs similar to the target product.

[0091] In some embodiments, the process of the server creating the feature comparison table includes the following steps e-Step I:

[0092] Step e: The server obtains the image feature vector of the reference similar appearance design and the image feature vector of the product to be put on the shelf;

[0093] In this process, the server obtains the image feature vector Vtarget of the product to be listed and the image feature vector Vref of the reference similar appearance design selected by the user according to the aforementioned step b1. Vtarget and Vref both include the image feature vectors Vtarget(ri) and Vref(ri) of several regions ri of the appearance image.

[0094] Step f: The server determines, based on the image feature vector of the reference similar design and the image feature vector of the product to be listed, at least one region where the product to be listed differs from the reference similar design, and the difference in the image feature vector of the at least one region;

[0095] The feature vector difference for each region is calculated as follows: △V(ri) = |Vtarget(ri) - Vref(ri)|.

[0096] Step g: The server determines the distinguishing feature comparison points from the at least one region based on the relationship between the difference in the image feature vectors and the preset difference.

[0097] The preset difference is ε. The distinguishing feature point R = {ri|ΔV(ri)≥ε} is determined based on the relationship between the image feature vector difference ΔV(ri) and the preset difference ε.

[0098] Step h: The server generates a textual description of the distinguishing feature comparison points based on the difference in image feature vectors of the distinguishing feature comparison points;

[0099] This involves determining the textual description of the specific differences in shape or pattern between the product to be listed and the reference similar design at the distinguishing feature point ri∈R, based on the difference in the image feature vectors of the distinguishing feature points ri∈R. For example, "At region r1, the edges of the product to be listed are smoother, while the edges of the reference design are sharper."

[0100] Step i: The server creates the feature comparison table based on the distinguishing feature comparison points and their corresponding text descriptions.

[0101] Specifically, a feature comparison table is generated based on the textual description of each distinguishing feature point ri∈R, forming a structured data display.

[0102] For example, see Figure 10 , Figure 10 The comparison table of product appearance features between the product to be marketed (heater) and a reference similar design (humidifier heater) provided in the embodiments of this application is based on... Figure 10 It can be seen that the reference design is somewhat similar to the overall appearance of the product to be listed, but there are also differences. In the front view, rear view, and left view, the positional relationship between the air distribution grille and the heater body of the product to be listed differs from the reference design, and this affects the frontal visual effect. In the rear view, the ventilation grille and groove positions of the product to be listed differ from the reference design, affecting the rear visual effect. In the top view, the top design of the product to be listed differs from the reference design; the product to be listed is a smooth flat surface, which affects the top visual effect.

[0103] As can be seen, in this embodiment, by extracting image features and calculating differences, areas where there are significant differences between the product to be listed and the reference design can be accurately identified. The image feature vector differences of the areas with significant differences are converted into textual descriptions, and a feature comparison table is constructed based on the textual descriptions. This improves the efficiency of product appearance comparison and allows users to understand the specific differences between different products at a glance.

[0104] In some embodiments, after displaying the feature comparison table on the product appearance compliance inspection interface, the method further includes: displaying a design avoidance control on the product appearance compliance inspection interface; receiving a trigger operation on the design avoidance control, creating a design avoidance request message, the design avoidance request message carrying the feature comparison table; sending the design avoidance request message to the server, the design avoidance request message instructing the server to generate a design avoidance appearance drawing based on the feature comparison table; receiving a design avoidance response message from the server, the design avoidance response message including the design avoidance appearance drawing; and displaying the design avoidance appearance drawing on the product appearance compliance inspection interface.

[0105] In this process, after the terminal device displays the feature comparison table on the product appearance compliance inspection interface, it displays the circumvention design control. When the user clicks the circumvention design control, the terminal device creates a circumvention design request message and sends it to the server. After receiving the circumvention design request message, the server generates a circumvention design appearance diagram based on the feature comparison table and sends the circumvention design appearance diagram back to the terminal device. After receiving the circumvention design appearance diagram, the terminal device displays the circumvention design appearance diagram on the product appearance compliance inspection interface.

[0106] Furthermore, the product compliance appearance inspection interface also displays an inspection report creation control. When the user clicks on the inspection report creation control, the terminal device generates an appearance inspection report based on the circumvented design appearance drawing and the feature comparison table. Alternatively, the terminal device can directly generate an appearance inspection report based on the feature comparison table, and the user can click to download the appearance inspection report.

[0107] Furthermore, the server can generate multiple circumvention design drawings based on different distinguishing feature comparison points in the feature comparison table and different adjustment methods for different distinguishing feature comparison points. Users can select one or more design drawings from multiple circumvention design drawings to generate an appearance inspection report.

[0108] In specific implementation, such as Figure 5 As shown, after the feature comparison table is displayed on the product appearance compliance inspection interface 3, the terminal device also displays the design avoidance control and the inspection report creation control. When the user clicks the design avoidance control, the design avoidance appearance drawing display area 35 is displayed on the product appearance compliance inspection interface 3, as shown. Figure 7 As shown, one or more circumvention design appearance drawings are displayed in the circumvention design appearance drawing display area 35, and in Figure 7 The product compliance testing interface 3 shown displays a test report creation control. After the user selects one or more circumvention design appearance drawings and clicks the test report creation control, the terminal device generates an appearance test report based on the selected one or more circumvention design appearance drawings and the feature comparison table. Figure 8 As shown. When the user clicks as... Figure 5 After the test report control is created as shown, it will display as follows: Figure 8 The interface shown.

[0109] As can be seen, in this embodiment, users can instantly create and send a design avoidance request message by triggering the design avoidance control. The server quickly processes and returns the design avoidance appearance drawing without having to switch to other platforms or tools, thus improving the compliance detection function of the e-commerce platform and enhancing the user experience.

[0110] In some embodiments, the process of the server creating the circumvention design appearance diagram includes the following steps j-k:

[0111] Step j: The server obtains the feature comparison table and the image feature vector of the product to be listed.

[0112] Step k: The server generates one or more appearance avoidance design text schemes based on the distinguishing feature comparison points in the feature comparison table and the image feature vector of the product to be listed.

[0113] The process of the server creating one or more appearance avoidance design text schemes specifically includes the following steps k1-k3:

[0114] Step k1: The server obtains the significance of the distinguishing feature comparison points in the feature comparison representation in the appearance design of the product to be listed.

[0115] The importance of distinguishing features can be determined based on the general attention span of consumers when viewing a product and the impact of these features on the overall appearance. Specifically, distinguishing features that consumers can clearly perceive are considered more important; the greater the difference in appearance compared to similar designs, the less likely they are to be identified as identical or similar. Distinguishing features that consumers may not easily perceive are considered less important, and no special requirements need to be placed on their appearance compared to similar designs. The impact of distinguishing features on the overall appearance can be determined by factors such as their relative position, size, shape, and color within the overall appearance. The importance of these features is determined by a combination of general attention span and impact.

[0116] Step k2: The server determines the adjustment range of the image feature vector of the distinguishing feature comparison point based on the importance of the feature, and the adjustment range is positively correlated with the importance of the feature.

[0117] Step k3: The server generates one or more appearance avoidance design text schemes based on the adjustment range and the image feature vector of the product to be listed.

[0118] Step 1: The server generates the appearance avoidance design drawing based on one or more appearance avoidance design text schemes.

[0119] The server can use graph generation models such as BigGAN to generate appearance evasion design diagrams.

[0120] As can be seen, in this embodiment, the server evaluates the importance of features and ensures that key features are prioritized for adjustment, thereby more effectively avoiding design similarity.

[0121] It should be noted that the specific implementation process of this embodiment can be found in the specific implementation process described in the above method embodiments, and will not be described again here.

[0122] With the above Figure 2 The embodiments shown are consistent; please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application, such as... Figure 9 As shown, the terminal device 10 includes a processor 91, a memory 93, a communication interface 92, and one or more programs 931, which are stored in the memory 93 and configured to be executed by the processor 91. The programs include methods for performing the methods described in the above embodiments.

[0123] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0124] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0127] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0130] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0131] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A product appearance compliance inspection method based on multimodal fusion, characterized in that, The method is applied to terminal devices of an e-commerce platform, the e-commerce platform further including a server, and the method includes: The system acquires the appearance image and text description information of the product to be listed uploaded by the user. The text description information includes the name, purpose, key points of the product appearance design, and a description of the key points of the product appearance design. An appearance inspection control is displayed on the product appearance compliance inspection interface. Upon detecting a trigger operation on the appearance inspection control, an appearance compliance inspection request message is created and sent to the server. The appearance compliance inspection request message includes the appearance image and the text description information, and is used to instruct the server to filter at least one similar appearance design from the appearance design database that has a similarity greater than a preset similarity to the product to be listed. The appearance design database includes appearance images and text description information of multiple appearance designs corresponding to the product category of the product to be listed. The system receives an appearance compliance inspection response message from the server, the appearance compliance inspection response message including the at least one similar appearance design; and displays the at least one similar appearance design, a similarity score for each similar appearance design, and a feature comparison control corresponding to each similar appearance design on the product appearance compliance inspection interface. If a trigger operation is detected on the feature comparison control of any one of the at least one similar appearance designs, a feature comparison request message is created and sent to the server. The feature comparison request message is used to instruct the server to generate a feature comparison table between the product to be put on the shelf and the reference similar appearance design. Receive a feature comparison response message from the server, the feature comparison response message including the feature comparison table; and display the feature comparison table on the product appearance compliance inspection interface; The process of determining at least one similar design specifically includes the following steps: obtaining the appearance image and text description information of the product to be listed, and obtaining the appearance design vector database of the appearance design database, wherein the appearance design vector database includes the product appearance fusion feature vectors of the multiple appearance designs; The process involves obtaining the shape and pattern features of the product's appearance image, as well as the product dimensions of the product, including planar and three-dimensional products. A second weighting relationship is determined based on the product dimensions. This second weighting relationship characterizes the importance of shape and pattern features in different product dimensions. For planar products, the design focus is on pattern features, with shape features as a secondary factor. For three-dimensional products, the design focus is on shape features, with pattern features as a secondary factor. An image feature vector is then obtained based on the second weighting relationship, the shape features, and the pattern features. A text feature vector is obtained based on the text description information of the product to be listed; a target product appearance fusion feature vector is determined based on the image feature vector, the text feature vector, and the first weight ratio relationship between the image feature vector and the text feature vector; a vector search is performed on the appearance design vector database based on the target product appearance fusion feature vector to obtain at least one product appearance fusion feature vector with a similarity greater than the preset similarity to the target product appearance fusion feature vector; the at least one similar appearance design is determined based on the appearance design corresponding to the at least one product appearance fusion feature vector. The method for determining the first weight ratio includes the following steps: obtaining all image features of the appearance image, including overall image features and local image features; determining text features based on the text description content; judging the overlap between image features and text features; if the overlap is greater than a preset threshold, the weight of the text features is reduced; if the overlap is less than the preset threshold, the weight of the text features is increased; and determining the first weight ratio based on the weights of the text features and image features.

2. The method according to claim 1, characterized in that, The method for determining the second weighting ratio includes the following steps: If the product dimension is the planar product, then the first weight of the shape feature is less than the second weight of the pattern feature, and the sum of the first weight and the second weight is 1; If the product dimension is the three-dimensional product, then the first weight is greater than the second weight.

3. The method according to claim 1, characterized in that, The process of creating the feature comparison table includes the following steps: Obtain the image feature vector of the reference similar appearance design and the image feature vector of the product to be put on the shelf; Based on the image feature vector of the reference similar appearance design and the image feature vector of the product to be put on the shelves, at least one region where the product to be put on the shelves differs from the reference similar appearance design and the image feature vector difference of the at least one region are determined; Based on the relationship between the difference in the image feature vector and a preset difference, distinguishing feature comparison points are determined from at least one region; The text description of the distinguishing feature comparison point is generated based on the difference in the image feature vector of the distinguishing feature comparison point; The feature comparison table is created based on the distinguishing feature comparison points and their corresponding textual descriptions.

4. The method according to claim 3, characterized in that, After displaying the feature comparison table on the product appearance compliance inspection interface, the method further includes: The product appearance compliance inspection interface displays the design avoidance controls; Upon receiving a trigger operation on the avoidance design control, an avoidance design request message is created, the avoidance design request message carrying the feature comparison table; Send the circumvention design request message to the server, the circumvention design request message being used to instruct the server to generate an circumvention design appearance diagram based on the feature comparison table; The server receives an evasion design response message, which includes the evasion design appearance diagram. The circumvention design appearance drawing is displayed on the product appearance compliance inspection interface.

5. The method according to claim 4, characterized in that, The process of creating the circumvention design appearance drawing includes the following steps: Obtain the feature comparison table and the image feature vector of the product to be listed; One or more appearance avoidance design text schemes are generated based on the distinguishing feature comparison points in the feature comparison table and the image feature vector of the product to be listed; and the avoidance design appearance image is generated based on the one or more appearance avoidance design text schemes.

6. The method according to claim 5, characterized in that, The creation process of the one or more appearance avoidance design text schemes specifically includes the following steps: The significance of the distinguishing feature comparison points in the feature comparison characterization in the appearance design of the product to be listed is obtained; The adjustment range of the image feature vector of the distinguishing feature comparison point is determined based on the importance of the feature, and the adjustment range is positively correlated with the importance of the feature. One or more appearance avoidance design text schemes are generated based on the adjustment range and the image feature vector of the product to be listed.

7. An e-commerce platform system, characterized in that, The e-commerce platform system includes a terminal device and a server, wherein the terminal device is used to execute the step instructions in the method as described in any one of claims 1-6.

Images