Image sending method and device, electronic equipment and computer readable medium

Abstract

Embodiments of the present disclosure disclose an image sending method, device, electronic equipment and computer readable medium. A specific implementation of the method comprises: extracting feature information of three different levels of a shelf image to obtain a first feature, a second feature and a third feature; performing feature enhancement processing on the first feature, the second feature and the third feature to obtain a first enhanced feature, a second enhanced feature and a third enhanced feature; performing semantic segmentation processing on the first enhanced feature, the second enhanced feature and the third enhanced feature to obtain a set of shelf effective area recognition information; performing target detection processing on the second enhanced feature and the third enhanced feature to obtain a set of item recognition information; and generating a shelf segmentation effect picture and an item detection effect picture. The implementation method simultaneously completes shelf layer area segmentation and item detection, reduces the memory occupancy rate in the recognition process while ensuring the item recognition accuracy, and improves the execution efficiency of item recognition.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to image transmission methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Item recognition refers to the technology of processing and analyzing images to identify various items. Before item recognition, it is usually necessary to use shelf images to locate shelf layers and items. Currently, the common methods for locating shelf layers and items are: using an image semantic segmentation model to locate the shelf layer area and using an object detection model to locate the item area.

[0003] However, when using the above method for item recognition, the following technical problems often arise:

[0004] Using two deep convolutional neural network models, an image semantic segmentation model and an object detection model, for shelf layer region localization and item localization respectively will result in high GPU memory or memory usage and low execution efficiency for shelf layer region localization and item localization. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide image transmission methods, apparatuses, electronic devices, and computer-readable media to address the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide an image transmission method, which includes: extracting feature information at three different levels from a shelf image using a feature extraction network to obtain a first feature, a second feature, and a third feature; performing feature enhancement processing on the first feature, the second feature, and the third feature based on a feature enhancement module to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature; performing semantic segmentation processing on the first enhanced feature, the second enhanced feature, and the third enhanced feature using a semantic segmentation branch to obtain a shelf effective area identification information set; performing target detection processing on the second enhanced feature and the third enhanced feature using a target detection branch to obtain an item identification information set; generating a shelf segmentation effect map and an item detection effect map based on the shelf effective area identification information set and the item identification information set; and sending the shelf segmentation effect map and the item detection effect map to a target terminal for display.

[0008] Secondly, some embodiments of this disclosure provide an image transmitting apparatus, comprising: a feature extraction module configured to extract feature information at three different levels of a shelf image using a feature extraction network to obtain a first feature, a second feature, and a third feature; a feature enhancement module configured to perform feature enhancement processing on the first feature, the second feature, and the third feature based on the feature enhancement module to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature; a semantic segmentation module configured to perform semantic segmentation processing on the first enhanced feature, the second enhanced feature, and the third enhanced feature using a semantic segmentation branch to obtain a shelf effective area identification information set; a target detection module configured to perform target detection processing on the second enhanced feature and the third enhanced feature using a target detection branch to obtain an item identification information set; an information recognition module configured to generate a shelf segmentation effect map and an item detection effect map based on the shelf effective area identification information set and the item identification information set; and a display module configured to send the shelf segmentation effect map and the item detection effect map to a target terminal for display.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above embodiments of this disclosure have the following beneficial effects: the image transmission method of some embodiments of this disclosure can reduce the memory usage rate during the recognition process and improve the execution efficiency of item recognition while ensuring the accuracy of the item recognition result. Specifically, the reason for the high memory usage rate and low execution efficiency is that the use of two deep convolutional neural network models makes the calculation process complex, resulting in high memory usage rate and low execution efficiency. Based on this, the image transmission method of some embodiments of this disclosure firstly extracts feature information of three different levels from the shelf image using a feature extraction network to obtain a first feature, a second feature, and a third feature. Then, based on the feature enhancement module, the first feature, the second feature, and the third feature are subjected to feature enhancement processing to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature. Thus, features of different scales are fused, increasing the contextual information of the features and effectively enhancing the semantic information of the features. Furthermore, the computational complexity is reduced by using depthwise separable convolution during the processing. Next, the first enhanced feature, the second enhanced feature, and the third enhanced feature are subjected to semantic segmentation processing using a semantic segmentation branch to obtain a set of effective shelf area recognition information. This is then used to generate a shelf segmentation effect image. Then, the object detection branch is used to perform object detection processing on the second and third enhanced features to obtain the item recognition information set. This is then used to generate the item detection result map, improving the detection rate. Next, based on the shelf effective area recognition information set and the item recognition information set, a shelf segmentation result map and an item detection result map are generated. Thus, shelf layer region segmentation and item localization are completed in this model. Finally, the shelf segmentation result map and item detection result map are sent to the target terminal for display. Therefore, the above embodiments of this disclosure can use only one deep convolutional neural network model to simultaneously complete the tasks of shelf layer region segmentation and item localization, ensuring the accuracy of item localization and item recognition results while reducing memory usage during item recognition and improving execution efficiency. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the image transmission method according to some embodiments of this disclosure;

[0014] Figure 2 This is a flowchart of some embodiments of the image transmission method according to the present disclosure;

[0015] Figure 3 These are schematic diagrams illustrating the structure of some embodiments of the image transmitting apparatus according to this disclosure;

[0016] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0020] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0021] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0022] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of an image transmission method according to some embodiments of this disclosure.

[0024] exist Figure 1In the application scenario, firstly, the computing device 101 can use the feature extraction network 102 to obtain feature information at three different levels of the shelf image, resulting in a first feature 103, a second feature 104, and a third feature 105. Next, the computing device 101 can perform feature enhancement processing on the first feature 103, the second feature 104, and the third feature 105 based on the feature enhancement module 106, resulting in a first enhanced feature 107, a second enhanced feature 108, and a third enhanced feature 109. Then, the computing device 101 can use the semantic segmentation branch 110 to perform semantic segmentation processing on the first enhanced feature 107, the second enhanced feature 108, and the third enhanced feature 109, generating a shelf segmentation effect image 112. Furthermore, the computing device 101 can use the object detection branch 111 to perform object detection processing on the second enhanced feature 108 and the third enhanced feature 109, generating a detection effect image 113. Finally, the computing device 101 can transmit the shelf segmentation effect image 112 and the detection effect image 113 to the target terminal 114 for display.

[0025] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0026] It should be understood that Figure 1 The number of computing devices shown is merely illustrative. Any number of computing devices can be used depending on implementation needs.

[0027] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of image transmission according to the present disclosure. The image transmission method includes the following steps:

[0028] Step 201: Use a feature extraction network to extract feature information at three different levels from the shelf image to obtain the first feature, the second feature, and the third feature.

[0029] In some embodiments, the execution subject of the image sending method (e.g. Figure 1The computing device 101 shown can use a feature extraction network to extract feature information at three different levels from a shelf image, obtaining a first feature, a second feature, and a third feature. The shelf image can be an image captured by a mobile phone or a camera installed around the shelf. The first feature can be a feature matrix of the first layer of the shelf image extracted using three bottleneck residual modules in the feature extraction network. The second feature can be a feature matrix of the second layer of the shelf image extracted using seven bottleneck residual modules in the feature extraction network. The third feature can be a feature matrix of the third layer of the shelf image extracted using fifteen bottleneck residual modules and one adjusting bottleneck residual module in the feature extraction network.

[0030] Optionally, the bottleneck residual module mentioned above may include two 1×1 convolutional layers and one 3×3 convolutional layer. The execution entity utilizes a feature extraction network to extract feature information from three different levels of the shelf image, obtaining a first feature, a second feature, and a third feature, which may include the following steps:

[0031] The first step is to use the three bottleneck residual modules in the above feature extraction network to extract the features of the first layer of the above shelf image, and obtain the first feature.

[0032] The second step involves using the seven bottleneck residual modules in the aforementioned feature extraction network to extract the features of the second layer of the shelf image, thus obtaining the second feature.

[0033] The third step involves using the fifteen bottleneck residual modules and one adjusted bottleneck residual module in the aforementioned feature extraction network to extract the features of the third layer of the shelf image, thus obtaining the third feature. Specifically, in the adjusted bottleneck residual module, the convolutional kernel is a 3×3 convolutional layer with a dilation rate of 2 and a stride of 1.

[0034] Step 202: Perform feature enhancement processing on the first feature, the second feature, and the third feature based on the feature enhancement module to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature.

[0035] In some embodiments, the execution entity may perform feature enhancement processing on the first feature, the second feature, and the third feature based on the feature enhancement module to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature. The first enhanced feature is the feature obtained after performing feature enhancement processing on the first feature. The second enhanced feature is the feature obtained after performing feature enhancement processing on the second feature. The third enhanced feature is the feature obtained after performing feature enhancement processing on the third feature.

[0036] Optionally, the feature enhancement module may include: a first convolutional layer, a second convolutional layer, and a third convolutional layer. The first convolutional layer is a 3×3 kernel. The second convolutional layer is a depthwise separable convolutional layer with a 5×5 kernel. The third convolutional layer is a 1×1 kernel. The execution entity performs feature enhancement processing on the first feature, the second feature, and the third feature based on the feature enhancement module to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature, which may include the following steps:

[0037] The first step is to downsample the first feature to obtain a first downsampled feature with the same width and height as the second feature. The downsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0038] The second step involves upsampling the third feature to obtain a third upsampled feature with the same width and height as the second feature. This upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0039] The third step involves using the first convolutional layer to perform convolution processing on the first downsampled feature, the second feature, and the third upsampled feature to obtain the first processed feature, the second processed feature, and the third processed feature.

[0040] The fourth step is to add the first processing feature, the second processing feature, and the third processing feature together to obtain the first output feature.

[0041] Fifth, the first output feature is convolved using the second convolutional layer to obtain the second output feature.

[0042] Step 6: Upsample the second output feature to obtain a second upsampled output feature with the same width and height as the first feature. The upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0043] Step 7: Concatenate the second upsampled output feature with the first feature to obtain the first merged feature. The concatenation can be a Concat operation.

[0044] Step 8: The first merged feature is convolved using the third convolutional layer to obtain the first enhanced feature.

[0045] Step 9: Concatenate the second output feature with the second feature to obtain the second merged feature. The concatenation can be done using the `Concat` directive.

[0046] Step 10: Use the third convolutional layer to perform convolution processing on the second merged feature to obtain the second enhanced feature.

[0047] Step 11: Downsample the second output feature to obtain a second downsampled output feature with the same width and height as the third feature. The downsampling method may include, but is not limited to, the following: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0048] Step 12: Concatenate the second downsampled output feature with the third feature to obtain the third merged feature. This concatenation can be done using the `Concat` directive.

[0049] Step 13: The third convolutional layer is used to perform convolution processing on the third merged feature to obtain the third enhanced feature.

[0050] Step 203: Use the semantic segmentation branch to perform semantic segmentation processing on the first enhanced feature, the second enhanced feature and the third enhanced feature to obtain the set of effective shelf area identification information.

[0051] In some embodiments, the aforementioned execution entity may utilize a semantic segmentation branch to perform semantic segmentation processing on the first enhanced feature, the second enhanced feature, and the third enhanced feature to obtain a set of shelf effective area identification information. This set of shelf effective area identification information may be the feature result ultimately output by the semantic segmentation branch.

[0052] Optionally, the aforementioned execution entity utilizes a semantic segmentation branch to perform semantic segmentation processing on the first enhanced feature, the second enhanced feature, and the third enhanced feature to obtain a set of effective shelf area identification information, which may include the following steps:

[0053] The first step is to upsample the third enhanced feature to obtain a second upsampled feature with the same width and height as the second enhanced feature. The upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0054] The second step is to concatenate the second upsampled feature with the second enhanced feature to obtain the second enhanced merged feature. This concatenation can be done using the `Concat` directive.

[0055] The third step involves using the first convolutional layer to perform convolution processing on the second enhanced and merged features to obtain the first semantic features.

[0056] The fourth step involves upsampling the first semantic feature to obtain a first upsampled feature with the same width and height as the first enhanced feature. This upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0057] Fifth, the first upsampled feature and the first enhanced feature are concatenated to obtain the first enhanced merged feature. The concatenation can be done using the `Concat` directive.

[0058] The sixth step involves using the first convolutional layer to perform convolution processing on the first enhanced and merged features to obtain the second semantic features.

[0059] Step 7: Upsample the second semantic feature to obtain a final semantic feature with the same width and height as the shelf image, and use the final semantic feature as the set of effective area identification information for the shelf. The upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0060] Step 204: Use the object detection branch to perform object detection processing on the second and third enhanced features to obtain the object recognition information set.

[0061] In some embodiments, the aforementioned execution entity may utilize the object detection branch to perform object detection processing on the second and third enhanced features to obtain an object identification information set. This object identification information set is the final feature result output by the object detection branch. The object detection branch may include, but is not limited to, the SSD model (Single Shot MultiBox Detector).

[0062] Optionally, the object detection branch may include a fourth convolutional layer, a fifth convolutional layer, and a sixth convolutional layer. The fourth convolutional layer may be a convolutional layer with a 3×3 kernel, a stride of 2, and padding of 1. The fifth convolutional layer may be a convolutional layer with a 1×1 kernel. The sixth convolutional layer may be a convolutional layer with a 3×3 kernel. The execution entity uses the object detection branch to perform object detection processing on the second and third enhancement features to obtain an object recognition information set, which may include the following steps:

[0063] The first step involves using the fourth convolutional layer to perform convolution processing on the third enhancement feature to obtain the fourth feature. This fourth feature has 1024 channels, a feature scale of 1 / 32 of the shelf image, and the padding can be padding.

[0064] The second step involves using the fifth convolutional layer to perform convolution processing on the third enhancement feature to obtain the fifth feature. The fifth feature has 512 channels and a feature scale of 1 / 16 of the shelf image.

[0065] The third step involves using the fifth convolution process described above to obtain the sixth feature. This sixth feature has 256 channels and a feature scale that is 1 / 8 of the shelf image.

[0066] The fourth step involves using the fifth convolutional layer to perform convolution processing on the fourth feature, resulting in a fourth processed feature with half the number of channels.

[0067] The fifth step involves upsampling the fourth processed feature to obtain a fourth upsampled feature with the same width and height as the fifth feature. This upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0068] The sixth step is to concatenate the fourth upsampled feature with the fifth feature to obtain the fourth merged feature. This concatenation can be done using the `Concat` directive.

[0069] Step 7: Use the sixth convolutional layer to perform convolution processing on the fourth merged feature to obtain the fourth updated feature.

[0070] The eighth step involves using the fifth convolutional layer to perform convolution processing on the fifth feature, resulting in a fifth processed feature with half the number of channels.

[0071] The ninth step involves upsampling the fifth processed feature to obtain a fifth upsampled feature with the same width and height as the sixth feature. This upsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0072] Step 10: Concatenate the fifth upsampled feature with the sixth feature to obtain the fifth merged feature. This concatenation can be done using the `Concat` directive.

[0073] In the eleventh step, the fifth merged feature is processed by convolution using the sixth convolutional layer to obtain the fifth updated feature.

[0074] Step 12: The fifth update feature is processed by convolution using the fifth convolutional layer to obtain a sixth processing feature with twice the number of channels as the fifth update feature.

[0075] Step 13: Downsample the sixth processed feature to obtain a second downsampled feature with the same width and height as the fifth updated feature. The downsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0076] Step fourteen: Concatenate the second downsampling feature with the fifth updated feature to obtain the sixth merged feature. This concatenation can be done using the `Concat` directive.

[0077] Step 15: Use the sixth convolutional layer to perform convolution processing on the sixth merged feature to obtain the sixth updated feature.

[0078] Step sixteen: The sixth update feature is convolved using the fifth convolutional layer to obtain a seventh processed feature with twice the number of channels as the sixth update feature.

[0079] Step seventeen: Downsample the aforementioned seventh processing feature to obtain a third downsampled feature with the same width and height as the aforementioned fourth updated feature. The downsampling can include, but is not limited to, the following methods: nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation.

[0080] Step 18: Concatenate the third downsampling feature with the fourth updated feature to obtain the seventh merged feature. This concatenation can be done using the `Concat` directive.

[0081] Step 19: Use the sixth convolutional layer to perform convolution processing on the seventh merged feature to obtain the seventh updated feature.

[0082] Step 20: The fifth, sixth, and seventh update features mentioned above are identified as item identification information to obtain the item identification information set.

[0083] Step 205: Based on the set of effective area identification information of the shelf and the set of item identification information, generate a shelf segmentation effect map and an item detection effect map.

[0084] In some embodiments, the executing entity can utilize the shelf effective area identification information set and the item identification information set to generate multiple bounding boxes of different sizes and aspect ratios, centered on each pixel, to obtain the shelf segmentation effect image and the item detection effect image. The shelf segmentation effect image is an image with shelf layer area segmentation result identifiers. The item detection effect image is an image with item detection result identifiers.

[0085] As an example, the above shelf segmentation effect diagram can be an identification diagram of the effective area of ​​the supermarket shelf layer, and the above item detection effect diagram can be an identification diagram of the items placed on the supermarket shelf layer.

[0086] Step 206: Send the shelf segmentation effect diagram and the item detection effect diagram to the target terminal for display.

[0087] In some embodiments, the executing entity can transmit the shelf segmentation effect diagram and the item detection effect diagram to a target terminal for display. The target terminal can be a terminal with display capabilities used for shelf image monitoring.

[0088] In some optional implementations of certain embodiments, the aforementioned execution entity may also send alarm notification information to the aforementioned target terminal in response to the shelf segmentation effect diagram and the item detection effect diagram. In practice, the aforementioned preset item quantity can be set according to actual conditions, and no specific limitation is made here.

[0089] Therefore, the shelf segmentation diagram and target detection diagram can be transmitted to the target terminal for display. This allows shelf managers to determine whether the placement of items complies with regulations based on the shelf segmentation diagram and target detection diagram, and thus enables them to replenish or adjust the items placed on the shelves in a timely manner.

[0090] The above embodiments of this disclosure have the following beneficial effects: the image transmission method of some embodiments of this disclosure can reduce the memory usage rate during the recognition process and improve the execution efficiency of item recognition while ensuring the accuracy of the item recognition result. Specifically, the reason for the high memory usage rate and low execution efficiency is that the use of two deep convolutional neural network models makes the calculation process complex, resulting in high memory usage rate and low execution efficiency. Based on this, the image transmission method of some embodiments of this disclosure firstly extracts feature information of three different levels from the shelf image using a feature extraction network to obtain a first feature, a second feature, and a third feature. Then, based on the feature enhancement module, the first feature, the second feature, and the third feature are subjected to feature enhancement processing to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature. Thus, features of different scales are fused, increasing the contextual information of the features and effectively enhancing the semantic information of the features. Furthermore, the computational complexity is reduced by using depthwise separable convolution during the processing. Next, the first enhanced feature, the second enhanced feature, and the third enhanced feature are subjected to semantic segmentation processing using a semantic segmentation branch to obtain a set of effective shelf area recognition information. This is then used to generate a shelf segmentation effect image. Then, the object detection branch is used to perform object detection processing on the second and third enhanced features to obtain the item recognition information set. This is then used to generate the item detection result map, improving the detection rate. Next, based on the shelf effective area recognition information set and the item recognition information set, a shelf segmentation result map and an item detection result map are generated. Thus, shelf layer region segmentation and item localization are completed in this model. Finally, the shelf segmentation result map and item detection result map are sent to the target terminal for display. Therefore, the above embodiments of this disclosure can use only one deep convolutional neural network model to simultaneously complete the tasks of shelf layer region segmentation and item localization, ensuring the accuracy of item localization and item recognition results while reducing memory usage during item recognition and improving execution efficiency.

[0091] Further reference Figure 3 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an image transmitting apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 3As shown, the image transmitting device 300 in some embodiments includes: a feature extraction module 301, a feature enhancement module 302, a semantic segmentation module 303, a target detection module 304, an information recognition module 305, and a display module 306. The system comprises the following modules: Feature Extraction Module 301, configured to extract feature information at three different levels from the shelf image using a feature extraction network, resulting in a first feature, a second feature, and a third feature; Feature Enhancement Module 302, configured to perform feature enhancement processing on the first, second, and third features based on the feature enhancement module, resulting in a first enhanced feature, a second enhanced feature, and a third enhanced feature; Semantic Segmentation Module 303, configured to perform semantic segmentation processing on the first, second, and third enhanced features using a semantic segmentation branch, resulting in a shelf effective area recognition information set; Target Detection Module 304, configured to perform target detection processing on the second and third enhanced features using a target detection branch, resulting in an item recognition information set; Information Recognition Module 305, configured to generate a shelf segmentation effect map and an item detection effect map based on the shelf effective area recognition information set and the item recognition information set; and Display Module 306, configured to send the shelf segmentation effect map and the item detection effect map to a target terminal for display.

[0093] It is understandable that the modules described in the device 300 are similar to those in the reference. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to device 300 and the modules contained therein, and will not be repeated here.

[0094] The following is for reference. Figure 4 It shows a schematic diagram of the structure of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0095] like Figure 4 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. Processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0096] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 4 Each box shown can represent a device or multiple devices as needed.

[0097] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0098] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0099] In some implementations, clients and servers may communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and may interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0100] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: extract feature information at three different levels from the shelf image using a feature extraction network to obtain a first feature, a second feature, and a third feature; perform feature enhancement processing on the first feature, the second feature, and the third feature based on a feature enhancement module to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature; perform semantic segmentation processing on the first enhanced feature, the second enhanced feature, and the third enhanced feature using a semantic segmentation branch to obtain a shelf effective area recognition information set; perform target detection processing on the second enhanced feature and the third enhanced feature using a target detection branch to obtain an item recognition information set; generate a shelf segmentation effect map and an item detection effect map based on the shelf effective area recognition information set and the item recognition information set; and send the shelf segmentation effect map and the item detection effect map to a target terminal for display.

[0101] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] The modules described in some embodiments of this disclosure can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a feature extraction module, a feature enhancement module, a semantic segmentation module, an object detection module, an information recognition module, and a display module. The names of these modules do not necessarily limit the module itself; for example, the display module may also be described as "a module for displaying a shelf segmentation effect image and an object detection effect image."

[0104] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

Claims

1. An image transmission method, comprising: The feature extraction network is used to extract feature information at three different levels from the shelf image, resulting in the first feature, the second feature, and the third feature; The feature enhancement module performs feature enhancement processing on the first feature, the second feature, and the third feature, fusing them to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature. The feature enhancement module includes a first convolutional layer, a second convolutional layer, and a third convolutional layer. The first convolutional layer is a 3×3 convolutional layer, the second convolutional layer is a depthwise separable convolutional layer with a 5×5 convolutional kernel, and the third convolutional layer is a 1×1 convolutional layer. The second convolutional layer is used to perform convolution processing on the first output feature to obtain the second output feature. The first output feature is obtained based on the first feature, the second feature, and the third feature. The second output feature is then used to redistribute the first feature, the second feature, and the third feature to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature. The first enhanced feature, the second enhanced feature, and the third enhanced feature are semantically segmented using a semantic segmentation branch to obtain a set of shelf effective area identification information. The second and third enhanced features are processed by the target detection branch to obtain a set of item recognition information. Based on the set of effective area identification information of the shelf and the set of item identification information, a shelf segmentation effect map and an item detection effect map are generated. The shelf segmentation effect diagram and the item detection effect diagram are sent to the target terminal for display.

2. The method according to claim 1, wherein, The feature extraction network is used to extract feature information at three different levels from the shelf image, resulting in a first feature, a second feature, and a third feature, including: The first feature is obtained by extracting the features of the first layer of the shelf image using the three bottleneck residual modules in the feature extraction network. The second feature is obtained by extracting the features of the second layer of the shelf image using the seven bottleneck residual modules in the feature extraction network. The third feature is obtained by extracting the features of the third layer of the shelf image using the fifteen bottleneck residual modules and one adjusted bottleneck residual module in the feature extraction network.

3. The method according to claim 2, wherein, In the adjusted bottleneck residual module, the convolutional kernel is 3×3, the dilation rate of the convolutional layer is 2, and the convolution stride is 1.

4. The method according to claim 1, wherein, The feature enhancement module performs feature enhancement processing on the first feature, the second feature, and the third feature to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature, including: The first feature is downsampled to obtain a first downsampled feature with the same width and height as the second feature; The third feature is upsampled to obtain a third upsampled feature with the same width and height as the second feature; The first convolutional layer is used to perform convolution processing on the first downsampled feature, the second feature, and the third upsampled feature to obtain the first processed feature, the second processed feature, and the third processed feature; The first processing feature, the second processing feature, and the third processing feature are added together to obtain the first output feature; The first output feature is processed by convolution using the second convolutional layer to obtain the second output feature; Using the second output feature, the first feature, the second feature, and the third feature are redistributed to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature.

5. The method according to claim 4, wherein, The step of using the second output feature to perform redistribution processing on the first feature, the second feature, and the third feature respectively to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature includes: The second output feature is upsampled to obtain a second upsampled output feature with the same width and height as the first feature; The second upsampled output feature is concatenated with the first feature to obtain the first merged feature; The first merged feature is processed by convolution using the third convolutional layer to obtain the first enhanced feature; The second output feature is concatenated with the second feature to obtain the second merged feature; The second merged feature is obtained by performing convolution processing on the third convolutional layer; The second output feature is downsampled to obtain a second downsampled output feature with the same width and height as the third feature; The second downsampled output feature is concatenated with the third feature to obtain the third merged feature; The third convolutional layer is used to perform convolution processing on the third merged feature to obtain the third enhanced feature.

6. The method according to claim 4, wherein, The semantic segmentation process, which utilizes semantic segmentation branches to perform semantic segmentation on the first enhanced feature, the second enhanced feature, and the third enhanced feature, yields a set of effective shelf area identification information, including: The third enhanced feature is upsampled to obtain a second upsampled feature with the same width and height as the second enhanced feature; The second upsampled feature is concatenated with the second enhanced feature to obtain the second enhanced merged feature; The first convolutional layer is used to perform convolution processing on the second enhanced merged feature to obtain the first semantic feature; The first semantic feature is upsampled to obtain a first upsampled feature with the same width and height as the first enhanced feature; The first upsampled feature is concatenated with the first enhanced feature to obtain the first enhanced merged feature; The first enhanced merged feature is processed by convolution using the first convolutional layer to obtain the second semantic feature; The second semantic feature is upsampled to obtain a final semantic feature with the same width and height as the shelf image, and the final semantic feature is used as the set of effective area identification information for the shelf.

7. The method according to claim 1, wherein, The target detection branch includes a fourth convolutional layer, a fifth convolutional layer, and a sixth convolutional layer. The fourth convolutional layer is a convolutional layer with a kernel size of 3×3, a stride size of 2, and padding of 1. The fifth convolutional layer is a convolutional layer with a kernel size of 1×1. The sixth convolutional layer is a convolutional layer with a kernel size of 3×3. as well as The object detection branch is used to perform object detection processing on the second and third enhanced features to obtain an object recognition information set, including: The third enhancement feature is convolved using the fourth convolutional layer to obtain the fourth feature, wherein the fourth feature has 1024 channels and the feature scale is 1 / 32 of the shelf image; The fifth convolutional layer is used to perform convolution processing on the third enhanced feature to obtain the fifth feature, wherein the number of channels of the fifth feature is 512 and the feature scale is 1 / 16 of the shelf image; The second enhanced feature is convolved using the fifth convolutional layer to obtain the sixth feature, wherein the sixth feature has 256 channels and the feature scale is 1 / 8 of the shelf image; The fourth feature is processed by convolution using the fifth convolutional layer to obtain a fourth processed feature with half the number of channels; The fourth processing feature is upsampled to obtain a fourth upsampled feature with the same width and height as the fifth feature; The fourth upsampled feature is concatenated with the fifth feature to obtain the fourth merged feature; The fourth merged feature is obtained by performing convolution processing on the sixth convolutional layer; The fifth feature is processed by convolution using the fifth convolutional layer to obtain a fifth processed feature with half the number of channels; The fifth processing feature is upsampled to obtain a fifth upsampled feature with the same width and height as the sixth feature; The fifth upsampled feature is concatenated with the sixth feature to obtain the fifth merged feature; The fifth merged feature is obtained by performing convolution processing on the sixth convolutional layer; The fifth convolutional layer is used to perform convolution processing on the fifth update feature to obtain a sixth processed feature with twice the number of channels as the fifth update feature; The sixth processing feature is downsampled to obtain a second downsampled feature with the same width and height as the fifth updated feature; The second downsampled feature is concatenated with the fifth updated feature to obtain the sixth merged feature; The sixth convolutional layer is used to perform convolution processing on the sixth merged feature to obtain the sixth updated feature; The sixth update feature is processed by convolution using the fifth convolutional layer to obtain a seventh processed feature with twice the number of channels as the sixth update feature. The seventh processing feature is downsampled to obtain a third downsampled feature with the same width and height as the fourth updated feature; The third downsampling feature is concatenated with the fourth updated feature to obtain the seventh merged feature; The seventh merged feature is obtained by performing convolution processing on the sixth convolutional layer; The fifth update feature, the sixth update feature, and the seventh update feature are determined as item identification information to obtain the item identification information set.

8. An image transmitting device, comprising: The feature extraction module is configured to use a feature extraction network to extract feature information at three different levels from the shelf image, resulting in the first feature, the second feature, and the third feature. A feature enhancement module is configured to perform feature enhancement processing on the first feature, the second feature, and the third feature based on the feature enhancement module, and fuse them to obtain a first enhanced feature, a second enhanced feature, and a third enhanced feature. The feature enhancement module includes a first convolutional layer, a second convolutional layer, and a third convolutional layer. The first convolutional layer is a convolutional layer with a 3×3 kernel, the second convolutional layer is a depthwise separable convolutional layer with a 5×5 kernel, and the third convolutional layer is a convolutional layer with a 1×1 kernel. Specifically, the second convolutional layer is used to perform convolution processing on the first output feature to obtain a second output feature. The first output feature is obtained based on the first feature, the second feature, and the third feature. The second output feature is then used to perform redistribution processing on the first feature, the second feature, and the third feature respectively to obtain the first enhanced feature, the second enhanced feature, and the third enhanced feature. The semantic segmentation module is configured to perform semantic segmentation processing on the first enhanced feature, the second enhanced feature and the third enhanced feature using a semantic segmentation branch to obtain a set of shelf effective area identification information. The target detection module is configured to perform target detection processing on the second enhanced feature and the third enhanced feature using the target detection branch to obtain a set of item recognition information; The information recognition module is configured to generate a shelf segmentation effect diagram and an item detection effect diagram based on the shelf effective area recognition information set and the item recognition information set; The display module is configured to send the shelf segmentation effect diagram and the item detection effect diagram to the target terminal for display.

9. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images