Commodity detection method, device, equipment, medium and program product

Abstract

The invention provides a commodity detection method which can be applied to the technical field of artificial intelligence. The commodity detection method comprises the steps that a real-time display image of a live broadcasting room at a preset moment is acquired, and the real-time display image comprises at least one display commodity; obtaining a commodity display image related to the marked commodity; image area comparison is carried out on the real-time display image and the commodity display image by using a commodity comparison model to obtain a comparison result, and the comparison result represents whether the real-time display image and the commodity display image comprise similar image areas; according to the comparison result, a commodity detection result is determined, and the commodity detection result comprises a first detection result representing whether the at least one displayed commodity comprises the labeled commodity or not. The invention further provides a commodity detection device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically to a commodity detection method, device, equipment, medium and program product. Background Art

[0002] With the development of the live broadcast e-commerce industry, live shopping is becoming more and more popular among users. Merchants display products to users through live broadcasts, and users can purchase products from the purchase link in the live broadcast room. Therefore, in order to ensure the user's purchasing experience, it is necessary to test the products displayed on the live broadcast room screen to ensure that the products displayed on the live broadcast room screen match the products displayed in the shopping link.

[0003] When detecting products displayed in a live broadcast room, it is usually necessary to use a positioning algorithm to locate the position of the product in the live broadcast room screen and then identify the product. Therefore, the accuracy of the positioning algorithm affects the accuracy of product identification, and for complex live broadcast room screens, it is difficult to accurately determine the position of the product in the live broadcast room screen through the positioning algorithm. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a commodity detection method, device, equipment, medium and program product.

[0005] According to a first aspect of the present disclosure, a commodity detection method is provided, comprising: obtaining a real-time display image of a live broadcast room at a preset time, wherein the real-time display image includes at least one displayed commodity; obtaining a commodity display image related to a marked commodity; using a commodity comparison model, performing image area comparison on the real-time display image and the commodity display image to obtain a comparison result, wherein the comparison result indicates whether the real-time display image and the commodity display image include similar image areas; determining a commodity detection result based on the comparison result, wherein the commodity detection result includes a first detection result indicating whether at least one of the displayed commodities includes the marked commodity.

[0006] According to an embodiment of the present disclosure, the above-mentioned product comparison model includes a first feature extraction network and a second feature extraction network; the above-mentioned product comparison model is used to perform image area comparison on the above-mentioned real-time display image and the above-mentioned product display image to obtain a comparison result, including: inputting the above-mentioned real-time display image into the first feature extraction network to obtain output first area image features, wherein the above-mentioned first area image features include the appearance features of each of the above-mentioned displayed products; inputting the above-mentioned product display image into the second feature extraction network to obtain output second area image features, wherein the above-mentioned second area image features include the appearance features of the above-mentioned marked products; and determining the above-mentioned comparison result according to the similarity between the above-mentioned first area image features and the above-mentioned second area image features.

[0007] According to an embodiment of the present disclosure, the above-mentioned product comparison model also includes an input layer, and the above-mentioned real-time display image and the above-mentioned product display image are input into the above-mentioned input layer in the form of an image pair; the above-mentioned method also includes: after the above-mentioned image pair is input into the above-mentioned input layer, determining a first identifier representing the above-mentioned real-time display image and a second identifier representing the above-mentioned product display image in the above-mentioned image pair, so as to input the above-mentioned real-time display image and the above-mentioned product display image into the above-mentioned first feature extraction network and the above-mentioned second feature extraction network respectively based on the above-mentioned first identifier and the above-mentioned second identifier.

[0008] According to an embodiment of the present disclosure, the first feature extraction network includes a feature extraction layer and a multi-head attention layer; the real-time display image is input into the first feature extraction network to obtain the output first region image feature, including: inputting the real-time display image into the feature extraction layer to obtain the output region scale feature, wherein the region scale feature is related to the size of the convolution kernel in the feature extraction layer; inputting the region scale feature into the multi-head attention layer to obtain the output first region image feature with region preference; wherein the region preference is related to the first weight information obtained by training the first feature extraction network, and the image region related to the displayed product in the first region image feature has a higher weight.

[0009] According to an embodiment of the present disclosure, the first area image feature and the second area image feature have the same multiple feature dimensions, and the product comparison model also includes a feature matching network; wherein the method also includes: using the feature matching network to calculate the sub-similarity between each group of sub-image features in the first area image feature and each group of sub-image features in the second area image feature, wherein each group of sub-image features includes a preset number of feature dimensions; and determining the similarity between the first area image feature and the second area image feature based on the sub-similarity.

[0010] According to an embodiment of the present disclosure, the method further includes: based on the second weight information of each group of sub-image features, and according to the sub-similarity, determining the similarity between the first area image features and the second area image features; wherein the second weight information is determined after training the feature matching network based on training samples, and the second weight information is used to characterize the feature preference for the feature dimension.

[0011] According to an embodiment of the present disclosure, the above-mentioned commodity detection result also includes a second detection result for characterizing that at least one of the above-mentioned displayed commodities belongs to the above-mentioned marked commodity; wherein, determining the commodity detection result based on the above-mentioned comparison result includes: determining the above-mentioned second detection result based on the similarity between the above-mentioned first area image feature and the above-mentioned second area image feature.

[0012] A second aspect of the present disclosure provides a commodity detection device, including: a first acquisition module, used to acquire a real-time display image of a live broadcast room at a preset time, wherein the real-time display image includes at least one displayed commodity; a second acquisition module, used to acquire a commodity display image related to a marked commodity; a comparison module, used to use a commodity comparison model to perform image area comparison on the real-time display image and the commodity display image to obtain a comparison result, wherein the comparison result indicates whether the real-time display image and the commodity display image include similar image areas; a detection module, used to determine a commodity detection result based on the comparison result, wherein the commodity detection result includes a first detection result indicating whether at least one of the displayed commodities includes the marked commodity.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] According to the embodiments of the present disclosure, by directly determining whether the real-time display image and the product display image include similar image areas, when the real-time display image and the product display image include similar image areas, it is determined that the displayed products include the labeled products, thereby eliminating the need to perform positioning operations in the live broadcast room screen and eliminating the need to use a large number of product samples to train the product recognition algorithm, thereby improving the detection speed and expanding the scope of applicable products. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1A diagram schematically illustrates an application scenario of a commodity detection method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of a commodity detection method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 3A A schematic diagram schematically shows a method for determining a comparison result according to a specific embodiment of the present disclosure;

[0021] Figure 3B A schematic diagram of determining a comparison result according to another specific embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A schematic diagram of a first feature extraction network according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A schematic diagram of a commodity comparison model according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 A structural block diagram of a commodity detection device according to an embodiment of the present disclosure is schematically shown; and

[0025] Figure 7 A block diagram of an electronic device suitable for implementing a commodity detection method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] When detecting goods in the live broadcast room screen, it is usually necessary to detect the position of the goods in the live broadcast room screen. However, the display position of the goods in different live broadcast room screens is not the same, so it is impossible to pre-set the position that needs to be detected in the live broadcast room screen. Usually, a positioning algorithm is used to determine the position of the goods in the live broadcast room screen, which causes the accuracy of the positioning algorithm to directly affect the accuracy of the product detection. The embodiment of the present disclosure directly compares the live broadcast room screen with the product display image in the shopping link. When the live broadcast room screen and the product display image in the shopping link include similar image areas, it is determined that the goods displayed in the live broadcast room screen include the goods in the shopping link, thereby eliminating the need to perform positioning operations in the live broadcast room screen and the need to train the product recognition algorithm using a large number of product samples.

[0031] An embodiment of the present disclosure provides a commodity detection method, including: obtaining a real-time display image of a live broadcast room at a preset time, wherein the real-time display image includes at least one displayed commodity; obtaining a commodity display image related to a marked commodity; using a commodity comparison model to perform image area comparison on the real-time display image and the commodity display image to obtain a comparison result, wherein the comparison result indicates whether the real-time display image and the commodity display image include similar image areas; determining a commodity detection result based on the comparison result, wherein the commodity detection result includes a first detection result indicating whether at least one displayed commodity includes the marked commodity.

[0032] Figure 1 The application scenario diagram of the commodity detection method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0034] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0037] It should be noted that the commodity detection method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the commodity detection device provided in the embodiment of the present disclosure can generally be set in the server 105. The commodity detection method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the commodity detection device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0039] The following will be based on Figure 1 The scene described by Figure 2~Figure 5 The commodity detection method of the disclosed embodiment is described in detail.

[0040] Figure 2 The flowchart of the commodity detection method according to the embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the commodity detection method of this embodiment includes operations S210 to S240.

[0042] In operation S210, a real-time display image of the live broadcast room at a preset time is obtained.

[0043] According to an embodiment of the present disclosure, the preset time can be a certain moment in the live broadcast room during the live broadcast process. The live broadcast room can be a live broadcast room that is currently broadcasting, or a historical live broadcast room after the live broadcast ends, without any restriction.

[0044] According to an embodiment of the present disclosure, the real-time display image includes at least one display commodity. Multiple preset moments can be set for each live broadcast room to obtain real-time images at each preset moment, and the real-time image including the display commodity is screened out from the multiple real-time images as the display image.

[0045] In operation S220, a product display image related to the marked product is acquired.

[0046] According to an embodiment of the present disclosure, the marked products may be products in a shopping link in the live broadcast room, specifically, may be products in a shopping link marked as “being explained”.

[0047] According to an embodiment of the present disclosure, the product display image may be a display image for the marked product in a shopping link of the marked product.

[0048] According to an embodiment of the present disclosure, when obtaining a product display image, a target shopping link related to the marked product can be determined from multiple shopping links in the live broadcast room, and then at least one display image related to the marked product can be selected from multiple display images of the target shopping link as the product display image. Specifically, the product display image can be the main display image of the product in the target shopping link.

[0049] In operation S230, the product comparison model is used to compare image regions of the real-time display image and the product display image to obtain a comparison result.

[0050] According to the embodiments of the present disclosure, the product comparison model can be used to compare image regions of real-time display images and product display images. Specifically, the real-time display image can be divided into multiple first sub-regions, and the product display image can be compared with the multiple first sub-regions respectively. The product display image can also be divided into multiple second sub-regions, and the real-time display image can be compared with the multiple second sub-regions respectively. Multiple first sub-regions and multiple second sub-regions can also be compared respectively, and there is no limitation on this.

[0051] According to an embodiment of the present disclosure, the comparison result indicates whether the real-time display image and the product display image include similar image regions. Specifically, the similar image region may be a first sub-region similar to the product display image, or a second sub-region similar to the real-time display image, or a similar first sub-region and second sub-region among a plurality of first sub-regions and a plurality of second sub-regions.

[0052] According to the embodiments of the present disclosure, since the product comparison model only focuses on whether the real-time display image and the product display image include similar image areas, and determines the comparison result by similarity, but does not focus on the specific display content of the real-time display image and the product display image, there is no need to locate the product from the real-time display image, nor is there any need to identify the product, thereby improving the product detection speed and expanding the scope of applicable products.

[0053] In operation S240, a commodity detection result is determined according to the comparison result.

[0054] According to an embodiment of the present disclosure, the commodity detection result includes a first detection result characterizing whether at least one displayed commodity includes a labeled commodity. When the comparison result characterizes that the real-time display image and the commodity display image include similar image areas, the first detection result is determined to be that the displayed commodity includes the labeled commodity. When the comparison result characterizes that the real-time display image and the commodity display image do not include similar image areas, the first detection result is determined to be that the displayed commodity does not include the labeled commodity.

[0055] According to an embodiment of the present disclosure, the commodity detection result may also include other detection results besides the first detection result, such as the similarity between the displayed commodity and the labeled commodity, etc., and there is no limitation on this.

[0056] According to the embodiments of the present disclosure, by directly determining whether the real-time display image and the product display image include similar image areas, when the real-time display image and the product display image include similar image areas, it is determined that the displayed products include the labeled products, thereby eliminating the need to perform positioning operations in the live broadcast room screen and eliminating the need to use a large number of product samples to train the product recognition algorithm, thereby improving the detection speed and expanding the scope of applicable products.

[0057] Figure 3A A schematic diagram of determining a comparison result according to a specific embodiment of the present disclosure is schematically shown.

[0058] like Figure 3A As shown, the first real-time display image 311 shows that the live broadcast staff is introducing the product XX mouthwash, and the product XX mouthwash is placed in the center of the table. The first product display image 321 includes the product XX mouthwash. The first real-time display image 311 and the first product display image 321 include a similar image area 330, and it is determined that at least one displayed product includes a marked product.

[0059] Figure 3B A schematic diagram of determining comparison results according to another specific embodiment of the present disclosure is schematically shown.

[0060] like Figure 3B As shown, the second real-time display image 312 shows that the live broadcast staff is introducing a product computer, which is placed on the left edge of the table. The second product display image 322 includes a product camera. The second real-time display image 312 and the second product display image 322 do not include similar image areas, and it is determined that at least one displayed product does not include a marked product.

[0061] According to an embodiment of the present disclosure, a product comparison model is used to perform image region comparison on a real-time display image and a product display image to obtain a comparison result, including: inputting the real-time display image into a first feature extraction network to obtain output first region image features; inputting the product display image into a second feature extraction network to obtain output second region image features; and determining the comparison result based on the similarity between the first region image features and the second region image features.

[0062] According to an embodiment of the present disclosure, the product comparison model includes a first feature extraction network and a second feature extraction network. Specifically, the first feature extraction network can be used to extract the appearance features of the displayed product from the real-time display image, and the second feature extraction network can be used to extract the appearance features of the labeled product from the product display image.

[0063] It should be noted that, in the process of extracting the real-time display image using the first feature extraction network, there is no need to determine the positions of multiple displayed goods in the real-time display screen, nor is there any need to determine the number of displayed goods included in the real-time display screen. It is only necessary to directly extract the real-time display screen.

[0064] According to an embodiment of the present disclosure, the first region image feature includes the appearance feature of each displayed commodity. Since the real-time display screen may include multiple displayed commodities, the extracted first region image feature may include the appearance features of multiple displayed commodities.

[0065] It should also be noted that, in the process of extracting the product display image using the second feature extraction network, there is no need to determine the position of the marked product in the product display screen, and it is only necessary to directly extract the product display screen.

[0066] According to an embodiment of the present disclosure, the second region image feature includes an appearance feature of the marked commodity.

[0067] According to the embodiments of the present disclosure, since the forms of commodity display in real-time display images and commodity display images are different, for example, displayed commodities may only occupy a small area in the real-time display image, while labeled commodities may occupy a larger area in the commodity display image. Therefore, in order to improve the accuracy of the first area image features and the second area image features, the first feature extraction network can be trained using historical real-time display images to obtain a first feature extraction network suitable for real-time display images, and the second feature extraction network can be trained using historical commodity display images to obtain a second feature extraction network suitable for commodity display images, so as to improve the accuracy of feature extraction, thereby improving the accuracy of commodity detection results.

[0068] According to an embodiment of the present disclosure, when determining the similarity between the first region image feature and the second region image feature, the similarity may be determined by direct calculation, such as calculating cosine similarity, or by using a feature matching network.

[0069] According to an embodiment of the present disclosure, the comparison result is obtained based on the similarity between the image features of the first region and the image features of the second region. Specifically, by setting a similarity threshold, when it is determined that the similarity between the image features of the first region and the image features of the second region is greater than or equal to a preset threshold, it is determined that the real-time display image and the product display image include similar image regions, and when it is determined that the similarity between the image features of the first region and the image features of the second region is less than a preset threshold, it is determined that the real-time display image and the product display image do not include similar image regions.

[0070] According to the embodiments of the present disclosure, by utilizing the first feature extraction network and the second feature extraction network to extract the real-time display image and the product display image respectively, relatively accurate first region image features and second region image features are obtained to better characterize the appearance features of the displayed products and the labeled products, obtain accurate comparison results, and thereby improve the accuracy of the product detection results.

[0071] According to an embodiment of the present disclosure, a product comparison model is used to perform image area comparison on a real-time display image and a product display image to obtain a comparison result, which also includes: after the image pair is input into an input layer, a first identifier representing the real-time display image and a second identifier representing the product display image in the image pair are determined, and based on the first identifier and the second identifier, the real-time display image and the product display image are respectively input into a first feature extraction network and a second feature extraction network.

[0072] According to an embodiment of the present disclosure, since the first feature extraction network is applicable to real-time display images, and the second feature extraction network is applicable to product display images, and the real-time display images and the product display images are simultaneously input into the product comparison model, in order to improve the accuracy of feature extraction, it is necessary to accurately input the real-time display images into the first feature extraction network, and the product display images into the second feature extraction network.

[0073] According to an embodiment of the present disclosure, the product comparison model further includes an input layer, and the real-time display image and the product display image are input into the input layer in the form of an image pair. Therefore, the input layer can be used to configure identifiers for the input real-time display image and the product display image, respectively, and the real-time display image and the product display image in the image pair are sent to the corresponding first feature extraction network and second feature extraction network respectively according to the identifiers.

[0074] According to an embodiment of the present disclosure, when real-time display images and product display images are input into an input layer in the form of image pairs, the order of the real-time display images and the product display images in the image pair can be set in advance. For example, the first image in the image pair can be set as the real-time display image, and the second image can be set as the product display image.

[0075] According to an embodiment of the present disclosure, when the position information of the real-time display image and the commodity display image in the image pair is determined, an identifier can be configured for the image in the image pair according to the position information. For example, a first identifier can be configured for the first image in the image pair to indicate that the first image is a real-time display image, and a second identifier can be configured for the second image in the image pair to indicate that the second image is a commodity display image.

[0076] According to an embodiment of the present disclosure, when the real-time display image and the commodity display image in the image pair can be distinguished by the first identifier and the second identifier, the image can be input into the corresponding feature extraction network through the identifier. Specifically, when the first identifier represents the real-time display image, the image with the first identifier is input into the first feature extraction network, and when the second identifier represents the commodity display image, the image with the second identifier is input into the second feature extraction network.

[0077] According to an embodiment of the present disclosure, by respectively configuring a first identifier and a second identifier for a real-time display image and a product display image in an image pair, and using the first identifier and the second identifier to input the image into a corresponding feature extraction network, the accuracy of feature extraction is improved, thereby improving the accuracy of product detection results.

[0078] According to an embodiment of the present disclosure, a real-time display image is input into a first feature extraction network to obtain an output first region image feature, including: inputting the real-time display image into a feature extraction layer to obtain an output region scale feature; inputting the region scale feature into a multi-head attention layer to obtain an output first region image feature with region preference.

[0079] According to an embodiment of the present disclosure, the first feature extraction network includes a feature extraction layer and a multi-head attention layer. In order to improve the accuracy of the first feature extraction network, a multi-head attention layer can be set after the feature extraction layer to use the multi-head attention layer to set a higher weight for the image area related to the displayed product.

[0080] According to an embodiment of the present disclosure, the feature extraction layer may be a convolution layer, which performs convolution processing on the real-time display image using a convolution kernel to obtain a regional scale feature. The regional scale feature is related to the size of the convolution kernel in the feature extraction layer, that is, the granularity of the regional scale feature is determined by the size of the convolution kernel.

[0081] According to an embodiment of the present disclosure, the region preference is related to the first weight information obtained by training the first feature extraction network, and the image region related to the displayed goods in the first region image feature has a higher weight.

[0082] According to an embodiment of the present disclosure, when the first feature extraction network is trained, a first feature extraction network with first weight information can be obtained. For example, the image area related to the displayed goods may be in the middle and lower area of ​​the real-time display screen, so the first weight information obtained by training can configure a higher weight for the middle and lower area of ​​the real-time display screen.

[0083] According to an embodiment of the present disclosure, the multi-head attention layer can adjust the weight of the convolution kernel through the first weight information. Specifically, the multi-head attention layer adjusts the attention weight of the convolution kernel through the first weight information, and then adjusts the weight of the convolution kernel according to the attention weight, thereby obtaining the first regional image feature with regional preference.

[0084] According to an embodiment of the present disclosure, a first region image feature with region preference is obtained through first weight information and a multi-head attention layer, thereby improving the representation capability of the first region image feature for the appearance features of displayed goods, thereby obtaining a more accurate comparison result.

[0085] Figure 4 A schematic diagram of a first feature extraction network according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 4 As shown, the first feature extraction network includes a feature extraction layer 41 and a multi-head attention layer 42.

[0087] According to an embodiment of the present disclosure, the real-time display image 10 is input to the feature extraction layer 41, and the regional scale feature 411 is output. The regional scale feature 411 is input to the multi-head attention layer 42, and the first regional image feature 11 is output.

[0088] According to an embodiment of the present disclosure, determining the similarity between the first region image feature and the second region image feature includes: using a feature matching network to calculate the sub-similarity between each group of sub-image features in the first region image feature and each group of sub-image features in the second region image feature; and determining the similarity between the first region image feature and the second region image feature based on the sub-similarity.

[0089] According to an embodiment of the present disclosure, the first region image feature and the second region image feature have the same multiple feature dimensions. Specifically, when the first region image feature includes 512 dimensions, the second region image feature also includes 512 dimensions.

[0090] According to an embodiment of the present disclosure, the product comparison model further includes a feature matching network. Specifically, the feature matching network can be used to compare each group of sub-image features in the first region image features with each group of sub-image features in the second region image features to obtain sub-similarity.

[0091] According to an embodiment of the present disclosure, each group of sub-image features includes a preset number of feature dimensions. For example, the sub-image features may include 8 dimensions, then the first image features include 64 groups of first sub-image features, and the second image features include 64 groups of second sub-image features.

[0092] According to an embodiment of the present disclosure, when calculating the sub-similarity between each group of sub-image features in the first region image features and each group of sub-image features in the second region image features, specifically, the first group of first sub-image features can be used to perform similarity calculations with 64 groups of second sub-image features to obtain 64 sub-similarity, and then the second group of first sub-image features can be used to perform similarity calculations with 64 groups of second sub-image features, until the 64th group of first sub-image features is used to perform similarity calculations with 64 groups of second sub-image features.

[0093] According to an embodiment of the present disclosure, when using sub-similarity to determine the similarity between the image features of the first area and the image features of the second area, a preset sub-similarity threshold can be set to determine whether there is a target sub-similarity greater than or equal to the preset sub-similarity threshold, and the similarity between the image features of the first area and the image features of the second area can be determined based on the number of target sub-similarity and the similarity value of each target sub-similarity.

[0094] According to an embodiment of the present disclosure, when it is determined that a target sub-similarity exists among a plurality of sub-similarity, it can be determined that a similar image region is included in the real-time display image and the product display image.

[0095] According to an embodiment of the present disclosure, the accuracy of the similarity is improved by comparing the first region image features and the second region image features using multiple feature dimensions.

[0096] Figure 5 A schematic diagram of a product comparison model according to an embodiment of the present disclosure is schematically shown.

[0097] like Figure 5 As shown, the product comparison model includes a first feature extraction network 510 , a second feature extraction network 520 and a feature matching network 530 .

[0098] According to an embodiment of the present disclosure, the image pair input to the product comparison model 500 includes a real-time display image 10 and a product display image 20. The real-time display image 10 is input to the first feature extraction network 510, feature extraction is performed, and the first region image feature 11 is output. The product display image 20 is input to the second feature extraction network 520, feature extraction is performed, and the second region image feature 21 is output. The first region image feature 11 and the second region image feature 21 are input to the feature matching network 530, and the similarity 30 is output. According to the similarity 30, the comparison result 40 is determined.

[0099] According to an embodiment of the present disclosure, determining the similarity between the first region image feature and the second region image feature further includes: determining the similarity between the first region image feature and the second region image feature according to the sub-similarity based on the second weight information of each group of sub-image features.

[0100] According to an embodiment of the present disclosure, the second weight information is determined after training a feature matching network based on training samples, and the second weight information is used to characterize feature preferences for feature dimensions.

[0101] According to an embodiment of the present disclosure, when training a feature matching network, a feature matching network with second weight information can be obtained. For example, a higher weight can be set for a sub-similarity related to a target feature dimension that may be used to characterize the appearance of a displayed product to characterize a feature preference for the target feature dimension.

[0102] According to an embodiment of the present disclosure, since the similarity calculation is performed in units of sub-image features, the second weight information of the sub-image features can be calculated based on the second weight information of each feature dimension in the sub-image features.

[0103] According to an embodiment of the present disclosure, after obtaining the sub-similarity between the first sub-image feature and the second sub-image feature, the sub-similarity can be weighted according to the first and second weight information corresponding to the first sub-image feature and the second weight information corresponding to the second sub-image feature to obtain a weighted sub-similarity, and then the similarity can be obtained based on multiple weighted sub-similarity.

[0104] According to an embodiment of the present disclosure, by using a feature matching network with feature preference to calculate the similarity between the first image region feature and the second image region feature, the accuracy of the similarity is improved.

[0105] According to an embodiment of the present disclosure, determining a commodity detection result according to the comparison result includes: determining a second detection result according to a similarity between an image feature of the first region and an image feature of the second region.

[0106] According to an embodiment of the present disclosure, the commodity detection result also includes a second detection result for characterizing that at least one displayed commodity is a marked commodity.

[0107] According to an embodiment of the present disclosure, when determining the second detection result based on the similarity, the second detection result can be determined based on multiple target sub-similarity related to the similarity. Since similar first sub-image features and second sub-image features can be determined through the target sub-similarity, similar image areas included in the real-time display image and the product display image can be determined.

[0108] According to an embodiment of the present disclosure, when it is determined that the first target image region in the real-time display image is similar to the second target image region in the product display image, a second detection result is determined based on the first target image region and the second target image region.

[0109] According to an embodiment of the present disclosure, by determining the second detection result based on the similarity, the positions of similar image areas in the real-time display image and the product display image are determined, thereby improving the richness of information contained in the product detection result.

[0110] Based on the above commodity detection method, the present disclosure also provides a commodity detection device. Figure 6 The device is described in detail.

[0111] Figure 6 The structural block diagram of a commodity detection device according to an embodiment of the present disclosure is schematically shown.

[0112] like Figure 6 As shown, the commodity detection device 600 of this embodiment includes a first acquisition module 610 , a second acquisition module 620 , a comparison module 630 and a detection module 640 .

[0113] The first acquisition module 610 is used to acquire a real-time display image of the live broadcast room at a preset time, wherein the real-time display image includes at least one displayed commodity. In one embodiment, the first acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0114] The second acquisition module 620 is used to acquire the commodity display image related to the marked commodity. In one embodiment, the second acquisition module 620 can be used to perform the operation S220 described above, which will not be described in detail here.

[0115] The comparison module 630 is used to compare the image regions of the real-time display image and the product display image using the product comparison model to obtain a comparison result, wherein the comparison result indicates whether the real-time display image and the product display image include similar image regions. In one embodiment, the comparison module 630 can be used to perform the operation S230 described above, which will not be described in detail herein.

[0116] The detection module 640 is used to determine a commodity detection result according to the comparison result, wherein the commodity detection result includes a first detection result indicating whether at least one of the displayed commodities includes the labeled commodity. In one embodiment, the detection module 640 can be used to perform the operation S240 described above, which will not be described in detail here.

[0117] According to an embodiment of the present disclosure, the product comparison model includes a first feature extraction network and a second feature extraction network. The comparison module 630 includes a first extraction submodule, a second extraction submodule and a comparison submodule.

[0118] The first extraction submodule is used to input the real-time display image into a first feature extraction network to obtain output first region image features, wherein the first region image features include appearance features of each of the displayed commodities.

[0119] The second extraction submodule is used to input the commodity display image into a second feature extraction network to obtain output second region image features, wherein the second region image features include appearance features of the marked commodity.

[0120] The comparison submodule is used to determine the comparison result according to the similarity between the first region image feature and the second region image feature.

[0121] According to an embodiment of the present disclosure, the product comparison model further includes an input layer, and the real-time display image and the product display image are input into the input layer in the form of an image pair. The comparison module 630 further includes an identification determination submodule.

[0122] The identification determination submodule is used to determine a first identification representing the real-time display image and a second identification representing the product display image in the image pair after the image pair is input into the input layer, so as to input the real-time display image and the product display image into the first feature extraction network and the second feature extraction network respectively based on the first identification and the second identification.

[0123] According to an embodiment of the present disclosure, the first feature extraction network includes a feature extraction layer and a multi-head attention layer. The first extraction submodule also includes a first extraction unit and an attention unit.

[0124] The first extraction unit is used to input the real-time display image into the feature extraction layer to obtain output regional scale features, wherein the regional scale features are related to the size of the convolution kernel in the feature extraction layer.

[0125] The attention unit is used to input the regional scale feature into the multi-head attention layer to obtain the output of the first regional image feature with regional preference; wherein the regional preference is related to the first weight information obtained by training the first feature extraction network, and the image area related to the displayed product in the first regional image feature has a higher weight.

[0126] According to an embodiment of the present disclosure, the first region image feature and the second region image feature have the same multiple feature dimensions, and the product comparison model further includes a feature matching network. The comparison module 630 further includes a similarity calculation submodule and a first determination submodule.

[0127] The similarity calculation submodule is used to calculate the sub-similarity between each group of sub-image features in the first area image features and each group of sub-image features in the second area image features using the feature matching network, wherein each group of sub-image features includes a preset number of feature dimensions.

[0128] The similarity determination submodule is used to determine the similarity between the first region image feature and the second region image feature according to the sub-similarity.

[0129] According to an embodiment of the present disclosure, the comparison module 630 further includes a second determination submodule.

[0130] The second determination submodule is used to determine the similarity between the first area image features and the second area image features based on the second weight information of each group of sub-image features and according to the sub-similarity; wherein the second weight information is determined after training the feature matching network based on training samples, and the second weight information is used to characterize the feature preference for the feature dimension.

[0131] According to an embodiment of the present disclosure, the commodity detection result further includes a second detection result for characterizing that at least one of the displayed commodities belongs to the marked commodity. The detection module 640 includes a result determination submodule.

[0132] The result determination submodule is used to determine the second detection result according to the similarity between the first region image feature and the second region image feature.

[0133] According to an embodiment of the present disclosure, any multiple modules of the first acquisition module 610, the second acquisition module 620, the comparison module 630 and the detection module 640 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 610, the second acquisition module 620, the comparison module 630 and the detection module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the first acquisition module 610 , the second acquisition module 620 , the comparison module 630 , and the detection module 640 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be executed.

[0134] Figure 7 A block diagram of an electronic device suitable for implementing a commodity detection method according to an embodiment of the present disclosure is schematically shown.

[0135] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0136] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0137] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0138] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0139] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0140] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the commodity detection method provided by the embodiment of the present disclosure.

[0141] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0142] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0143] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0144] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0145] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0147] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A commodity detection method, comprising: Acquire a real-time display image of a live broadcast room at a preset time, wherein the real-time display image includes at least one displayed commodity; Obtain product display images related to the marked product; Using the product comparison model, performing image region comparison on the real-time display image and the product display image to obtain a comparison result, wherein the comparison result indicates whether the real-time display image and the product display image include similar image regions; A commodity detection result is determined according to the comparison result, wherein the commodity detection result includes a first detection result indicating whether at least one of the displayed commodities includes the marked commodity.

2. The method according to claim 1, wherein: The product comparison model includes a first feature extraction network and a second feature extraction network; the use of the product comparison model to compare the image regions of the real-time display image and the product display image to obtain a comparison result includes: Inputting the real-time display image into a first feature extraction network to obtain output first region image features, wherein the first region image features include appearance features of each of the displayed commodities; Inputting the commodity display image into a second feature extraction network to obtain output second region image features, wherein the second region image features include appearance features of the marked commodity; and The comparison result is determined according to the similarity between the first region image feature and the second region image feature.

3. The method according to claim 2, wherein: The product comparison model further includes an input layer, wherein the real-time display image and the product display image are input into the input layer in the form of an image pair; The method further comprises: After the image pair is input into the input layer, a first identifier representing the real-time display image and a second identifier representing the product display image in the image pair are determined, so that based on the first identifier and the second identifier, the real-time display image and the product display image are respectively input into the first feature extraction network and the second feature extraction network.

4. The method according to claim 2, wherein: The first feature extraction network includes a feature extraction layer and a multi-head attention layer; The step of inputting the real-time display image into a first feature extraction network to obtain output first region image features includes: Inputting the real-time display image into the feature extraction layer to obtain output regional scale features, wherein the regional scale features are related to the size of the convolution kernel in the feature extraction layer; Inputting the regional scale feature into the multi-head attention layer to obtain the output first regional image feature with regional preference; The region preference is related to first weight information obtained by training the first feature extraction network, and the image region related to the displayed product in the first region image feature has a higher weight.

5. The method according to claim 2, wherein: The first region image feature and the second region image feature have the same multiple feature dimensions, and the product comparison model further includes a feature matching network; wherein the method further includes: Utilizing the feature matching network, calculating the sub-similarity between each group of sub-image features in the first region image features and each group of sub-image features in the second region image features, wherein each group of sub-image features includes a preset number of feature dimensions; The similarity between the first region image feature and the second region image feature is determined according to the sub-similarity.

6. The method according to claim 5, further comprising: Based on the second weight information of each group of sub-image features, and according to the sub-similarity, determining the similarity between the first region image features and the second region image features; The second weight information is determined after training the feature matching network based on training samples, and the second weight information is used to characterize the feature preference for the feature dimension.

7. The method according to claim 2, wherein: The commodity detection result also includes a second detection result for characterizing that at least one of the displayed commodities belongs to the labeled commodity; wherein determining the commodity detection result according to the comparison result includes: The second detection result is determined according to the similarity between the first region image feature and the second region image feature.

8. A live broadcast room commodity detection device, comprising: A first acquisition module is used to acquire a real-time display image of a live broadcast room at a preset time, wherein the real-time display image includes at least one display commodity; A second acquisition module is used to acquire a commodity display image related to the marked commodity; A comparison module, configured to compare image regions of the real-time display image and the product display image using a product comparison model to obtain a comparison result, wherein the comparison result indicates whether the real-time display image and the product display image include similar image regions; The detection module is used to determine a commodity detection result according to the comparison result, wherein the commodity detection result includes a first detection result indicating whether at least one of the displayed commodities includes the marked commodity.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images