Image processing method, device and storage medium

Abstract

The embodiments of the present application provide an image processing method, device and storage medium. In some embodiments of the present application, a convolution operation is performed on at least one target image using a hardware-based convolution device to obtain at least one shared feature map; candidate frame areas are extracted from at least one shared feature map respectively; a pooler only performs pooling processing on the candidate frame areas of the shared feature map whose number of candidate frame areas is greater than a set threshold, and directly skips the shared feature map whose number is less than or equal to the set threshold; a software-hardware combined image processing architecture is provided, which uses limited hardware resources to quickly perform convolution and pooling operations on the image, thereby improving the efficiency of image processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the continuous development of technologies such as the Internet and e-commerce, more and more image data need to be processed. Convolutional Neural Networks (CNN) are widely used in computer vision for tasks such as classification, detection or segmentation.

[0003] Currently, convolutional neural networks implemented using pure software solutions are less efficient in image processing. Summary of the invention

[0004] Multiple aspects of the present application provide an image processing method that fully utilizes existing software and hardware resources to optimize the image processing process and improve the efficiency of image processing.

[0005] The present application provides an image processing method, including:

[0006] Inputting at least one target image into a hardware-based convolutional device;

[0007] Performing feature extraction on at least one target image to obtain at least one shared feature map;

[0008] Extracting candidate box regions for at least one shared feature map respectively, and providing at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0009] A target shared feature map is identified from at least one shared feature map in which the number of candidate frame areas is greater than a set threshold, and in the pooler, the candidate frame areas of the target shared feature map are pooled to obtain a pooled feature map.

[0010] The embodiment of the present application also provides an image processing device, including: a processor, a memory, and a convolution device and a pooling device implemented based on hardware;

[0011] The memory is used to store one or more computer instructions;

[0012] The processor is configured to execute the one or more computer instructions to:

[0013] Inputting at least one target image into a hardware-based convolutional device;

[0014] Performing feature extraction on at least one target image to obtain at least one shared feature map;

[0015] Extracting candidate box regions for at least one shared feature map respectively, and providing at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0016] A target shared feature map is identified from at least one shared feature map in which the number of candidate frame areas is greater than a set threshold, and the pooler performs pooling processing on the candidate frame areas of the target shared feature map to obtain a pooled feature map.

[0017] The embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to perform the following actions:

[0018] Inputting at least one target image into a hardware-based convolution device to extract features of the at least one target image, thereby obtaining at least one shared feature map;

[0019] Extracting candidate box regions for at least one shared feature map respectively, and providing at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0020] A target shared feature map is identified from at least one shared feature map in which the number of candidate frame areas is greater than a set threshold, and in the pooler, the candidate frame areas of the target shared feature map are pooled to obtain a pooled feature map.

[0021] The present application also provides an image processing method, including:

[0022] Acquire a video clip containing a code scanning action, wherein the video clip includes multiple images;

[0023] Selecting at least one target image containing a person and a product from the plurality of images;

[0024] Inputting at least one target image into a hardware-based convolutional device;

[0025] Performing feature extraction on at least one target image to obtain at least one shared feature map;

[0026] Extracting candidate box regions for at least one shared feature map respectively, and providing at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0027] Identifying a target shared feature map in which the number of candidate frame areas is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map;

[0028] The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

[0029] The embodiment of the present application also provides an image processing device, including: a processor, a memory, and a convolution device and a pooling device implemented based on hardware;

[0030] The memory is used to store one or more computer instructions;

[0031] The processor is configured to execute the one or more computer instructions to:

[0032] Acquire a video clip containing a code scanning action, wherein the video clip includes multiple images;

[0033] Selecting at least one target image containing a person and a product from the plurality of images;

[0034] Inputting at least one target image into a hardware-based convolutional device;

[0035] Performing feature extraction on at least one target image to obtain at least one shared feature map;

[0036] Extracting candidate box regions for at least one shared feature map respectively, and providing at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0037] Identifying a target shared feature map in which the number of candidate frame areas is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map;

[0038] The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

[0039] The embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to perform the following actions:

[0040] Acquire a video clip containing a code scanning action, wherein the video clip includes multiple images;

[0041] Selecting at least one target image containing a person and a product from the plurality of images;

[0042] Inputting at least one target image into a hardware-based convolutional device;

[0043] Performing feature extraction on at least one target image to obtain at least one shared feature map;

[0044] Extracting candidate box regions for at least one shared feature map respectively, and providing at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0045] Identifying a target shared feature map in which the number of candidate frame areas is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map;

[0046] The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

[0047] In some embodiments of the present application, a hardware-based convolution device is used to perform a convolution operation on at least one target image to obtain at least one shared feature map; candidate frame areas are extracted from at least one shared feature map; the pooler only performs pooling processing on the candidate frame areas of the shared feature map whose number of candidate frame areas is greater than a set threshold, and directly skips the shared feature maps whose number is less than or equal to the set threshold; and a software-hardware combined image processing architecture is provided, which utilizes limited hardware resources to quickly perform convolution and pooling operations on images, thereby improving the efficiency of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 A flowchart of an image processing method provided for an exemplary embodiment of the present application;

[0050] Figure 2 The embodiment of the present application provides a top-down design framework;

[0051] Figure 3 A flowchart of an image processing method provided for an exemplary embodiment of the present application;

[0052] Figure 4 A schematic diagram of a flow chart of a pooler performing pooling processing on a candidate frame area of ​​a target shared feature map provided by an exemplary embodiment of the present application;

[0053] Figure 5 A flowchart of another image processing method provided for an exemplary embodiment of the present application;

[0054] Figure 6a A method flow chart of another image processing method provided for an exemplary embodiment of the present application;

[0055] Figure 6bAn image processing method for scanning a product for checkout provided in an exemplary embodiment of the present application;

[0056] Figure 7 A schematic diagram of the structure of an image processing device provided for an exemplary embodiment of the present application;

[0057] Figure 8 A schematic structural diagram of an image processing device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0059] Today, consumers are increasingly using self-service devices to conduct transactions with economic entities. Self-service devices come in a variety of sizes and are used for a variety of purposes, such as fast food restaurants, pharmacies, and stores.

[0060] At present, a self-service checkout system has emerged, in which consumers complete the entire checkout process by themselves. Consumers who use the self-service checkout system need to scan all the purchased goods by themselves and pay at the settlement terminal. Specifically, consumers use scanning devices to scan the barcode of the goods, and the system finds the corresponding price of the goods in the system's product information database according to the barcode, and calculates the total purchase amount of the goods based on the price information, and the consumer pays at the settlement terminal.

[0061] When consumers are scanning barcodes to settle their purchases, there may be situations where the barcode scanning fails or the barcode is not scanned manually. However, there is no method to supervise the above situations, which causes losses to merchants.

[0062] In view of the fact that existing consumers cannot detect in time the situation that the barcode scanning is unsuccessful or the barcode is not scanned manually during the process of scanning and settling the goods, it is necessary to provide a method for timely detecting the situation that the barcode scanning is unsuccessful or the barcode is not scanned manually. In some embodiments of the present application, a convolution operation is performed on at least one target image using a hardware-based convolution device to obtain at least one shared feature map; candidate frame areas are extracted from at least one shared feature map respectively; the pooler only performs pooling processing on the candidate frame areas of the shared feature map whose number of candidate frame areas is greater than a set threshold, and directly skips the shared feature maps whose number is less than or equal to the set threshold; a software-hardware combined image processing architecture is provided to quickly perform convolution and pooling operations on images using limited hardware resources to improve the efficiency of image processing.

[0063] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0064] In this embodiment, a convolutional neural network is deployed on an image processing device for image processing. The image processing device first uses a hardware-based convolution device to perform a convolution operation on at least one target image to obtain at least one shared feature map; then, candidate frame areas are extracted from the at least one shared feature map respectively; finally, the pooler only performs pooling processing on the candidate frame areas of the shared feature map whose number of candidate frame areas is greater than a set threshold, and directly skips the shared feature maps whose number is less than or equal to the set threshold, thereby providing a software-hardware combined image processing architecture, which uses limited hardware resources to quickly perform convolution and pooling operations on images, thereby improving the efficiency of image processing.

[0065] In the above embodiment, the convolutional neural network includes an input layer, a hidden layer and a fully connected layer; the hidden layer includes a convolutional layer, a pooling layer and a fully connected layer. In this embodiment, the number of convolutional layers, pooling layers and fully connected layers is not limited, and the convolutional layers and pooling layers can be implemented as convolution devices and poolers based on FPGA or CPLD. The convolution device mainly extracts features from the input target image to obtain a shared feature map; the pooler mainly compresses the candidate frame area. On the one hand, the candidate frame area is compressed, and on the other hand, the candidate frame area is compressed into a feature map of the same size, which is convenient for the subsequent classification network to perform image classification operations.

[0066] Figure 1 Flow chart of an image processing method provided by an exemplary embodiment of the present application. Figure 1 As shown, in this embodiment, the image processing device needs to identify the video stream in advance in order to identify the video segment containing the code scanning action. Figure 1As shown, the video stream collected by the visual sensor is decoded. On the one hand, the decoded video stream is sent to a convolution device used for tracking and detecting people and goods. The convolution device convolves each frame of the video stream to output whether each frame contains information about people and goods. On the other hand, optical flow detection is performed on the decoded video stream. The optical flow detection can detect the movement of objects (mainly people and goods). Then, according to whether each frame of the image output by the convolution device used for tracking and detecting people and goods contains information about people and goods, combined with the optical flow detection result of the video stream, the original video segment containing the code scanning action can be determined. Then, the video segment to be detected is obtained according to the original video segment containing the code scanning action.

[0067] Further, continue to refer to Figure 1 After the scanning action is detected, the video clips to be detected are respectively subjected to image detection of missed scanning and direct bagging without scanning. If the classification result is that there is a situation of missed scanning or direct bagging without scanning, the alarm operation is performed.

[0068] In the embodiment of the present application, a convolutional neural network can be designed first, and the convolutional neural network is used to perform convolution and pooling operations on the target image; based on this process, the embodiment of the present application provides a top-down design framework 100, such as Figure 2 As shown, the design framework 100 includes several layers: front-end design 11, graph optimization 12, execution system 13, instruction calculation 14 and hardware support 15.

[0069] See also Figure 2 , in the front-end design 11 layer, you can use some open source convolutional neural network development frameworks or software, such as Tensorflow, Caffe or Mxnet, to design the computational graph of the convolutional neural network. The design of the computational graph includes but is not limited to the following aspects: the definition and construction of the computational graph, recording the nodes, operation types and connection relationships of the computational graph, and converting the format of the computational graph into a graph representation in a system-defined format, etc.

[0070] Continue to see Figure 2 , in the graph optimization layer 12, the designed computation graph can be optimized. Optionally, the optimization of the computation graph includes performing unified, device-independent optimization on the computation graph, such as elimination of dead nodes, node fusion, etc.; and also includes performing device-dependent optimization on the computation graph, such as instruction transformation, device allocation, convolution, and pooling.

[0071] Continue to see Figure 2 , at the execution system 13 layer, it mainly provides the central control core required for the operation of the convolutional neural network, and is responsible for the overall management of all affairs, including the thread pool, memory pool, data cache involved in the operation of the convolutional neural network, and the management and scheduling of various devices in the hardware device layer.

[0072] Continue to see Figure 1 , in the instruction calculation layer 14, it is mainly responsible for defining the instruction calculation process of the relevant hardware devices in the hardware support layer required for the operation of the convolutional neural network, such as whether it is compiled execution or interpreted execution. For example, the process of feature extraction using a hardware-implemented convolution device can abstract a matrix-matrix multiplication operation, including three processes starting from image data, convolution kernel loading, executing calculations, and outputting results. Therefore, in the instruction calculation layer, it is also possible to design matrices, vectors, convolutions, storage, and other instructions related to convolution and pooling related to these three processes.

[0073] Continue to see Figure 2 In the hardware support layer, it mainly provides the hardware devices such as CPU 151, GPU 152, convolution device 153a or pooler 153b implemented based on hardware (FPGA 153) required for the operation of convolutional neural network.

[0074] For example, in the process of convolution or pooling using a convolution device or pooler, the convolution device or pooler can decide the instruction execution mode of the convolution or pooling operator according to the scheduling of the CPU or GPU, the constraints of its own hardware resources and the minimum off-chip storage (for example, the off-chip storage can be DDR) access request principle, and perform pipeline scheduling on the instructions. The instructions here are designed by the instruction calculation layer.

[0075] based on Figure 2 The schematic diagram of the soft-hard combination framework is shown in FIG. Figure 3 The flowchart of an image processing method provided by the exemplary embodiment of the present application is as follows. The image processing method is not limited to the above-mentioned soft-hard combination framework, and other types of soft-hard combination frameworks can also be designed according to requirements, such as Figure 3 As shown, the method includes:

[0076] S301: Inputting at least one target image into a convolution device based on hardware implementation;

[0077] S302: extracting features from at least one target image to obtain at least one shared feature map;

[0078] S303: Extract candidate box regions for at least one shared feature map respectively, and provide the at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0079] S304: Identify a target shared feature map whose number of candidate box regions is greater than a set threshold from at least one shared feature map, and perform pooling processing on the candidate box regions of the target shared feature map in a pooler to obtain a pooled feature map.

[0080] In this embodiment, the image processing device that executes the method can be implemented by a terminal device with certain computing and processing capabilities, such as a self-service checkout device with video acquisition and code scanning functions installed in a supermarket or shopping mall, but it is not limited to this. The above method can also be implemented by the cooperation between the self-service checkout device and the server deployed in the cloud. For example, the self-service checkout device obtains the video stream collected by the visual sensor, and based on the optical flow algorithm, it intercepts the video segment containing the code scanning action from the video stream as the video segment to be detected, and uploads the video segment to be detected to the server, which executes Figure 3 Alternatively, the self-service checkout device obtains the video stream collected by the visual sensor, and based on the optical flow algorithm, extracts the video segment containing the code scanning action from the video stream as the video segment to be detected, and the self-service checkout device executes Figure 3 The steps of the method are described.

[0081] If the above method is implemented by the self-service checkout device and the server deployed in the cloud, the server can provide data support, computing services and some management services for the self-service checkout device. In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center and other server devices. Among them, the composition of the server device mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type. The server can include one website server or multiple website servers. In this embodiment, the self-service checkout device and the server establish a communication connection through wireless or wired. Optionally, the self-service checkout device can establish a communication connection with the server using WIFI, Bluetooth, infrared and other communication methods, or the self-service checkout device can also establish a communication connection with the server through a mobile network. Among them, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE) and 5G.

[0082] In this embodiment, a convolution device based on hardware implementation is used to extract features of the target image to obtain a shared feature map. The present application does not limit the architecture of the convolution device. In an optional embodiment, the convolution device includes a central control unit, an image cache unit, a convolution calculation unit, an intermediate result temporary storage unit, an input interface unit and an output buffer unit; first, the central control unit inputs the external target image and the convolution template coefficient of the target image through the input interface unit, and then the central control unit provides the input target image to the image cache unit of the convolution device; then, the central control unit provides the target image and the convolution template coefficient to the convolution calculation unit for multi-layer convolution to obtain a shared feature map; wherein the intermediate data generated during the convolution process is stored in the intermediate result temporary storage unit; finally, the shared feature map is output through the output buffer unit for subsequent processing of the shared feature map. wherein, the shared feature map reflects the different features of the target image extracted by the convolution device, and the convolution layer is composed of convolution kernels with learnable parameters; the width and length of the convolution kernel can be changed, and the depth must be consistent with the number of channels of the input layer. For example, if the input image is 32*32*3, the size of a convolution kernel is 5*5*3, and the sliding step is 1. If there are 6 5*5*3, a 28*28*6 shared feature map can be generated. The number of channels in the output layer is the same as the number of convolution kernels.

[0083] In this embodiment, after obtaining the shared feature map, the candidate frame area is extracted from the shared feature map. One achievable method is to use a region generation algorithm for the first shared feature map to determine multiple candidate regions to be determined from the first shared feature map, wherein the first shared feature map is any one of at least one shared feature map; the region greater than or equal to the set probability threshold among the multiple candidate regions to be determined is used as the candidate frame area of ​​the first shared feature map; and the number of candidate frame areas of the first shared feature map is counted. For example, the RPN algorithm is used to extract candidate frame areas from the shared feature map to obtain multiple candidate regions to be determined; a region greater than or equal to the set probability threshold is selected from the multiple candidate regions to be determined as the candidate frame area of ​​the first shared feature map, and the number of candidate frame areas is counted. It should be noted that the embodiment of the present application does not limit the value of the set probability threshold, and the set probability threshold can be adjusted according to actual conditions.

[0084] In the above embodiment, multiple convolvers and multiple poolers can be set in the FPGA or CPLD to perform parallel processing on multiple target images, so as to simultaneously obtain feature maps after pooling of multiple target images, fully utilize the parallelism of the FPGA or CPLD, and improve image processing efficiency.

[0085] In the above embodiment, the shared feature map and the candidate frame areas corresponding to the shared feature map and the number of candidate frame areas are stored in the memory of the image processing device, and the processor of the image processing device controls the shared feature map and the number of candidate frame areas to be provided to the pooler based on hardware implementation to perform pooling processing on the candidate frame areas. Optionally, the processor of the image processing device dynamically writes the number of candidate frame areas of multiple shared feature maps, the height and width of the output frame, and the number of channels processed in parallel into the cache corresponding to the pooler, the pooler parses the metadata, and writes the number of candidate frame areas into the register corresponding to the pooler, and the pooler can read the number of candidate frame areas in the register in real time.

[0086] In this embodiment, the pooler first determines whether the number of candidate box areas is greater than a set threshold. The pooler only performs pooling processing on the candidate box areas of the target shared feature map whose number of candidate box areas is greater than the set threshold, and directly skips the candidate box areas of the target shared feature map whose number of candidate box areas is less than or equal to the set threshold. For example, if the pooler reads that the number of candidate box areas of the current shared feature map is 0, the current shared feature map is not pooled, and the result that no candidate box area is detected is directly output, and the pooling processing of the current shared feature map is stopped; if the pooler reads that the number of candidate box areas of the current shared feature map is greater than 0, the candidate box areas of the current target shared feature map are pooled to obtain a pooled feature map. It should be noted that the present application does not limit the set threshold, and the set threshold can be adjusted according to actual conditions.

[0087] The image processing method of the embodiment of the present application calculates the number of candidate frame areas of at least one target image in real time through a compiler based on software implementation. When the number of candidate frame areas of the shared feature map is 0, the pooler directly stops the subsequent pooling operation, thereby improving the image processing efficiency through the collaboration of software and hardware. In addition, the compiler determines the table structure according to the sampling rate in advance, calculates the coordinates of the N integer pixel points closest to the interpolation point coordinates and the N interpolation coefficients around the interpolation point, calculates the number of candidate frame areas, outputs the height and width of the frame, and writes the above configuration information into the register corresponding to the pooler.

[0088] In the above embodiment, before the pooler pools the candidate frame area of ​​the shared feature map, in order to improve its own image processing performance, the pooler can set a feature map size threshold to limit the size of the input target shared feature map. When the size of the target shared feature map is too large, the target shared map can be segmented to be smaller than the feature map size threshold supported by the pooler. An optional embodiment is that the pooler determines whether the size of the target shared feature map is greater than the feature map size threshold supported by the pooler; if so, the target shared feature map is segmented into multiple feature map segments, and the size of each feature map segment is smaller than the feature map size threshold; the pooler performs pooling processing on the multiple feature map segments to obtain the pooling result of the target shared feature map. Among them, the segmentation of the target shared feature map can be randomly segmented along the depth direction, width direction or height direction until the size of each feature map segment after segmentation is smaller than the feature map size threshold. It should be noted that the embodiment of the present application does not limit the feature map size threshold, and the device hardware can adjust the feature map size threshold according to its own device parameters.

[0089] In the above embodiment, after obtaining the pooled feature map, the pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result. Among them, the type of at least one classifier can be set according to the different scenarios, and the corresponding number and type of classifiers can be set. For example, in the supermarket self-service checkout scenario, the user may miss the code or directly bag the code without scanning it during the code scanning process. A first classifier and a second classifier can be set on the image processing device, wherein the first classifier is used to detect whether there is a situation where the code is missed in the target image; the second classifier is used to detect whether there is a situation where the code is directly bagged without scanning it in the target image, and output the detection result of whether there is a missed code or direct bagging without scanning it.

[0090] In the above embodiment, after obtaining the classification result, the image processing device can perform a corresponding reminder operation. For example, in a supermarket self-service checkout scenario, if the classification results of the first classifier and the second classifier are that there is a situation where the code is missed or the goods are directly bagged without scanning the code, a warning operation is performed.

[0091] Based on the description of the above implementations, Figure 4 A schematic diagram of a flow chart of a pooler performing pooling processing on a candidate frame area of ​​a target shared feature map provided by an exemplary embodiment of the present application. Figure 3 As shown, the pooling process includes:

[0092] S401: Obtain a shared feature map and a candidate frame area of ​​the shared feature map to be pooled;

[0093] S402: Determine whether the number of candidate box regions of the shared feature map is 0; if it is 0, execute step S403; if not, execute step S404:

[0094] S403: Stop the pooling operation on the shared feature map;

[0095] S404: The pooler determines whether the size of the shared feature map is greater than the feature map size threshold supported by the pooler; if so, execute step S405; if not, execute step S406;

[0096] S405: dividing the shared feature map into a plurality of feature map segments, wherein the size of each feature map segment is smaller than a feature map size threshold;

[0097] S406: Start the vector calculation unit to perform pooling processing on the candidate box area to obtain a pooled feature map.

[0098] In this embodiment, the table entry structure is determined in advance according to the sampling rate, the coordinates of the N integer pixel points closest to the interpolation point coordinates and the N interpolation coefficients around the interpolation point are calculated, the number of candidate box areas is calculated, the height and width of the box are output, and the above configuration information is written into the register corresponding to the pooler.

[0099] The pooler first determines whether the size of the shared feature map is greater than the feature map size threshold supported by the pooler. If the pooler reads that the number of candidate frame areas of the current shared feature map is 0, the current shared feature map is not pooled, and the result that no candidate frame area can be detected is directly output, and the pooling process of the current shared feature map is stopped. If the pooler reads that the number of candidate frame areas of the current shared feature map is greater than 0, the pooler determines whether the size of the shared feature map is greater than the feature map size threshold supported by the pooler; if so, the shared feature map is divided into multiple feature map segments, and the size of each feature map segment is less than the feature map size threshold, and then the candidate frame area of ​​the current target shared feature map is pooled to obtain the pooled feature map; if not, the candidate frame area of ​​the current target shared feature map is directly pooled to obtain the pooled feature map.

[0100] Based on the description of the above embodiments, Figure 5 FIG. 1 is a flow chart of another image processing method provided by an exemplary embodiment of the present application. Figure 5 As shown, the method includes:

[0101] S501: Obtain a video clip to be convolved, where the video clip includes multiple images;

[0102] S502: Acquire at least one target image to be convolved from multiple images;

[0103] S503: Inputting at least one target image into a convolution device based on hardware implementation;

[0104] S504: extracting features from at least one target image to obtain at least one shared feature map;

[0105] S505: Extract candidate box regions for at least one shared feature map respectively, and provide the at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0106] S506: identifying a target shared feature map whose number of candidate box regions is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate box regions of the target shared feature map in a pooler to obtain a pooled feature map;

[0107] S507: Send the pooled feature map to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

[0108] Regarding the implementation of each step of this embodiment, please refer to the description of the corresponding parts of the aforementioned embodiments, which will not be repeated here.

[0109] The image processing method provided in the embodiment of the present application can be applied to various scenarios such as video processing and image recognition. In different application scenarios, the process of the image processing method will vary with the changes in the scenario. The following is an example of a self-service checkout in a shopping mall:

[0110] At present, there are consumer checkout systems in shopping malls and supermarkets. In this system, consumers complete the entire checkout process by themselves. Consumers who use the self-service checkout system need to scan all the purchased goods by themselves and pay at the settlement terminal. Specifically, consumers use scanning devices to scan the barcode of the goods. The system finds the price of the goods in the system's product information database according to the barcode, and calculates the total purchase amount of the goods based on the price information. The consumer pays at the settlement terminal. For example, users can install the corresponding shopping app on their mobile phones, scan the QR code on the product through the scanning function of the App, obtain the price, quantity and other information of the product, and then complete the payment online, which can improve payment efficiency and avoid wasting time in queues.

[0111] In the above scenario, users may miss scanning the code or directly bag the goods without scanning the code during the scanning process. A video acquisition module can be installed on the image processing device to collect the video stream of users scanning the code to pay in supermarkets or shopping malls, and then detect the missed scanning and direct bagging without scanning the code on the collected video stream. After detecting the missed scanning or direct bagging without scanning the code, a warning operation is performed to prompt the user to pay for the unsuccessful payment. Missing the code scan means that the user has initiated the scanning action but has not actually scanned the product information. The direct bagging without scanning the code here means, for example, that the user habitually bags the goods without scanning the code. The "bagging" here is in a broad sense, including various ways for users to store goods, such as boxing, packing, and packing in their own shopping carts.

[0112] Based on the above shopping mall self-service checkout scenario, Figure 6a A flowchart of another image processing method provided by an exemplary embodiment of the present application. Figure 6a As shown, the method includes:

[0113] S601: Collect a video clip to be convolved, where the video clip includes multiple images;

[0114] S602: selecting at least one target image containing a target object from the multiple images according to the displacement of pixels in the multiple images;

[0115] S603: Inputting at least one target image into a convolution device based on hardware implementation;

[0116] S604: Extract features from at least one target image to obtain at least one shared feature map;

[0117] S605: Extract candidate box regions for at least one shared feature map respectively, and provide the at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0118] S606: identifying a target shared feature map whose number of candidate box regions is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate box regions of the target shared feature map in a pooler to obtain a pooled feature map;

[0119] S607: Input the pooled feature map into the first classifier and the second classifier respectively, and output the detection result of whether there is a missed code scan or a code is directly bagged without being scanned; wherein the first classifier is used to detect whether there is a missed code scan in the target image; the second classifier is used to detect whether there is a code is directly bagged without being scanned in the target image.

[0120] In this embodiment, the image processing device that executes the method can be implemented by a terminal device with certain computing and processing capabilities, such as a self-service checkout device with video acquisition and code scanning functions installed in a supermarket or shopping mall.

[0121] In this embodiment, a visual sensor is provided on the image processing device, and the field of view of the visual sensor can cover the physical space where the user performs the code scanning action. The visual sensor collects a video stream of the user's code scanning process.

[0122] In this embodiment, after acquiring the video segment to be detected, the image processing device can upload the video segment to be detected to the server for storage. After responding to the viewing request initiated by the viewing terminal, the server sends the video segment to be detected to the viewing terminal for the user to view.

[0123] In this embodiment, a convolution device based on hardware implementation is used to extract features of the target image to obtain a shared feature map. The present application does not limit the architecture of the convolution device. In an optional embodiment, the convolution device includes a central control unit, an image cache unit, a convolution calculation unit, an intermediate result temporary storage unit, an input interface unit and an output buffer unit; first, the central control unit inputs the external target image and the convolution template coefficient of the target image through the input interface unit, and then the central control unit provides the input target image to the image cache unit of the convolution device; then, the central control unit provides the target image and the convolution template coefficient to the convolution calculation unit for multi-layer convolution to obtain a shared feature map; wherein, the intermediate data generated during the convolution process is stored in the intermediate result temporary storage unit; finally, the shared feature map is output through the output buffer unit for subsequent processing of the shared feature map.

[0124] In this embodiment, after obtaining the shared feature map, the candidate frame area is extracted from the shared feature map. One achievable method is to use a region generation algorithm for the first shared feature map to determine multiple candidate regions to be determined from the first shared feature map, wherein the first shared feature map is any one of at least one shared feature map; the region greater than or equal to the set probability threshold among the multiple candidate regions to be determined is used as the candidate frame area of ​​the first shared feature map; and the number of candidate frame areas of the first shared feature map is counted. For example, the RPN algorithm is used to extract candidate frame areas from the shared feature map to obtain multiple candidate regions to be determined; a region greater than or equal to the set probability threshold is selected from the multiple candidate regions to be determined as the candidate frame area of ​​the first shared feature map, and the number of candidate frame areas is counted. It should be noted that the embodiment of the present application does not limit the value of the set probability threshold, and the set probability threshold can be adjusted according to actual conditions.

[0125] In the above embodiment, multiple convolvers and multiple poolers can be set in the FPGA or CPLD to perform parallel processing on multiple target images, so as to simultaneously obtain feature maps after pooling of multiple target images, fully utilize the parallelism of the FPGA or CPLD, and improve image processing efficiency.

[0126] In the above embodiment, the shared feature map and the candidate frame areas corresponding to the shared feature map and the number of candidate frame areas are stored in the memory of the image processing device, and the processor of the image processing device controls the shared feature map and the number of candidate frame areas to be provided to the pooler based on hardware implementation to perform pooling processing on the candidate frame areas. Optionally, the processor of the image processing device dynamically writes the number of candidate frame areas of multiple shared feature maps, the height and width of the output frame, and the number of channels processed in parallel into the cache corresponding to the pooler, the pooler parses the metadata, and writes the number of candidate frame areas into the register corresponding to the pooler, and the pooler can read the number of candidate frame areas in the register in real time.

[0127] In this embodiment, the pooler first determines whether the number of candidate box areas is greater than a set threshold. The pooler only performs pooling processing on the candidate box areas of the target shared feature map whose number of candidate box areas is greater than the set threshold, and directly skips the candidate box areas of the target shared feature map whose number of candidate box areas is less than or equal to the set threshold. For example, if the pooler reads that the number of candidate box areas of the current shared feature map is 0, the current shared feature map is not pooled, and the result that no candidate box area is detected is directly output, and the pooling processing of the current shared feature map is stopped; if the pooler reads that the number of candidate box areas of the current shared feature map is greater than 0, the candidate box areas of the current target shared feature map are pooled to obtain a pooled feature map. It should be noted that the present application does not limit the set threshold, and the set threshold can be adjusted according to actual conditions.

[0128] The image processing method of the embodiment of the present application calculates the number of candidate frame areas of at least one target image in real time through a compiler based on software implementation. When the number of candidate frame areas of the shared feature map is 0, the pooler directly stops the subsequent pooling operation, thereby improving the image processing efficiency through the collaboration of software and hardware. In addition, the compiler determines the table structure according to the sampling rate in advance, calculates the coordinates of the N integer pixel points closest to the interpolation point coordinates and the N interpolation coefficients around the interpolation point, calculates the number of candidate frame areas, outputs the height and width of the frame, and writes the above configuration information into the register corresponding to the pooler.

[0129] In the above embodiment, before the pooler pools the candidate frame area of ​​the shared feature map, in order to improve its own image processing performance, the pooler can set a feature map size threshold to limit the size of the input target shared feature map. When the size of the target shared feature map is too large, the target shared map can be segmented to be smaller than the feature map size threshold supported by the pooler. An optional embodiment is that the pooler determines whether the size of the target shared feature map is greater than the feature map size threshold supported by the pooler; if so, the target shared feature map is segmented into multiple feature map segments, and the size of each feature map segment is smaller than the feature map size threshold; the pooler performs pooling processing on the multiple feature map segments to obtain the pooling result of the target shared feature map. Among them, the segmentation of the target shared feature map can be randomly segmented along the depth direction, width direction or height direction until the size of each feature map segment after segmentation is smaller than the feature map size threshold. It should be noted that the embodiment of the present application does not limit the feature map size threshold, and the device hardware can adjust the feature map size threshold according to its own device parameters.

[0130] After obtaining the pooled feature map, the feature maps of the same size are respectively sent to the first classifier and the second classifier to output the detection results of whether there is a missed code scan or a direct bagging without scanning the code. Among them, the first classifier is used to detect whether there is a missed code scan in the target image; the second classifier is used to detect whether there is a direct bagging without scanning the code in the target image. Among them, other types of classifiers can also be set to detect other situations in the self-service checkout process, for example, a third classifier is set to detect normal payment situations.

[0131] If the classification result shows that the code is not scanned or the bag is directly packed without scanning the code, a warning operation will be performed. The warning operation includes but is not limited to the following situations:

[0132] Scenario 1: An indicator light is provided on the image processing device, and the indicator light on the image processing device is controlled to emit a light signal;

[0133] Scenario 2: A speaker is provided on the image processing device, and the speaker on the image processing device is controlled to play a warning sound signal.

[0134] Combined with the description of the above embodiments, Figure 6b An image processing method for scanning a product code for checkout is provided in an exemplary embodiment of the present application. Figure 6b As shown, the method includes:

[0135] S611: Acquire a video clip containing a code scanning action, where the video clip includes multiple images;

[0136] S612: Select at least one target image containing a person and a product from the multiple images;

[0137] S613: Inputting at least one target image into a convolution device based on hardware implementation;

[0138] S614: Extract features from at least one target image to obtain at least one shared feature map;

[0139] S615: Extract candidate box regions for at least one shared feature map respectively, and provide the at least one shared feature map and the number of candidate box regions thereof to a pooler based on hardware implementation;

[0140] S616: identifying a target shared feature map whose number of candidate box regions is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate box regions of the target shared feature map in a pooler to obtain a pooled feature map;

[0141] S617: Send the pooled feature map to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

[0142] In a method embodiment of the image processing method of the present application, a convolution operation is performed on at least one target image using a hardware-based convolution device to obtain at least one shared feature map; candidate frame areas are extracted from at least one shared feature map respectively; a pooler only performs pooling processing on the candidate frame areas of the shared feature map whose number of candidate frame areas is greater than a set threshold, and directly skips the shared feature maps whose number is less than or equal to the set threshold; and a software-hardware combined image processing architecture is provided, which utilizes limited hardware resources to quickly perform convolution and pooling operations on images, thereby improving the efficiency of image processing.

[0143] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 601 to 606 can be device A; for another example, the execution subject of steps 601 to 603 can be device A, and the execution subject of steps 604 to 606 can be device B; and so on.

[0144] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as S301, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0145] Figure 7 The following is a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application. Figure 7 As shown, the device includes: a memory 701, a processor 702, a convolver 705 and a pooler 706 based on hardware implementation, and also includes necessary components such as a communication component 703, a power component 704, and additional components such as an audio component 707 and a display 708.

[0146] The memory 701 is used to store computer programs and can be configured to store various other data to support operations on the image processing device, such as instructions for any application program or method operating on the server.

[0147] Memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0148] The communication component 703 is used to establish a communication connection with other devices.

[0149] The processor 702 can execute computer instructions stored in the memory 701 to: input at least one target image into a convolution device based on hardware implementation; perform feature extraction on at least one target image to obtain at least one shared feature map; extract candidate frame areas for at least one shared feature map respectively, and provide at least one shared feature map and the number of its candidate frame areas to a pooler based on hardware implementation; identify a target shared feature map whose number of candidate frame areas is greater than a set threshold from at least one shared feature map, and perform pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map.

[0150] Optionally, the processor 702 may also be configured to: send the pooled feature map to at least one classifier based on hardware implementation for classification processing.

[0151] Optionally, before inputting at least one target image into a hardware-based convolution device to perform feature extraction on at least one target image, the processor 702 can also be used to: collect a video clip to be convolved, where the video clip includes multiple images; and select at least one target image containing a target object from the multiple images based on the displacement of pixels in the multiple images.

[0152] Optionally, at least one classifier includes: a first classifier and a second classifier; wherein the first classifier is used to detect whether there is a situation where the code is missed in the target image; the second classifier is used to detect whether there is a situation where the code is directly bagged without being scanned in the target image; the processor 702 can also be used to: if the classification result is that the code is missed or the code is directly bagged without being scanned, then execute a warning operation.

[0153] Optionally, when the processor 702 extracts candidate frame areas for at least one shared feature map respectively, it is specifically used to: for the first shared feature map, use a region generation algorithm to determine multiple pending candidate areas from the first shared feature map, wherein the first shared feature map is any one of the at least one shared feature map; take an area among the multiple pending candidate areas that is greater than or equal to a set probability threshold as a candidate frame area for the first shared feature map; and count the number of candidate frame areas for the first shared feature map.

[0154] Optionally, when the processor 702 provides the number of at least one shared feature map and its candidate frame area to a pooler based on hardware implementation, it is specifically used to: write the number of at least one shared feature map and its candidate frame area into a register corresponding to the pooler.

[0155] Optionally, when the processor 702 performs pooling processing on the candidate frame area of ​​the target shared feature map, it is specifically used to: determine whether the size of the target shared feature map is greater than the feature map size threshold supported by the pooler; if so, divide the target shared feature map into multiple feature map segments, and the size of each feature map segment is less than the feature map size threshold; perform pooling processing on the multiple feature map segments to obtain the pooling result of the target shared feature map.

[0156] Optionally, when the processor 702 collects the video clip to be convolved, it is specifically used to: collect the video clip to be convolved using the video sensor on the image processing device; the image processing device is also provided with an indicator light and a speaker, and when the processor 702 performs a warning operation, it includes at least one of the following: controlling the indicator light on the image processing device to emit a light signal; controlling the speaker on the image processing device to play a warning sound signal.

[0157] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 2 Each step in the method embodiment.

[0158] Figure 8 The following is a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application. Figure 8As shown, the device includes: a memory 801, a processor 802, a convolver 805 and a pooler 806 based on hardware implementation, and also includes necessary components such as a communication component 803, a power component 804, and additional components such as an audio component 807 and a display 808.

[0159] The memory 801 is used to store computer programs and can be configured to store various other data to support operations on the image processing device, such as instructions for any application or method used to operate on the server.

[0160] Memory 801 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0161] The communication component 803 is used to establish a communication connection with other devices.

[0162] The processor 802 can execute the computer instructions stored in the memory 801 to: obtain a video clip containing a code scanning action, wherein the video clip includes multiple images; select at least one target image containing a person and a product from the multiple images; and input the at least one target image into a convolution device based on hardware implementation;

[0163] Perform feature extraction on at least one target image to obtain at least one shared feature map; extract candidate frame areas for at least one shared feature map respectively, and provide at least one shared feature map and the number of its candidate frame areas to a pooler based on hardware implementation; identify a target shared feature map whose number of candidate frame areas is greater than a set threshold from at least one shared feature map, and perform pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map; send the pooled feature map to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

[0164] Optionally, the at least one classifier includes: a first classifier and a second classifier; wherein the first classifier is used to detect whether there is a situation where the code is missed in the target image; the second classifier is used to detect whether there is a situation where the code is directly bagged without being scanned in the target image. When the processor 802 sends the pooled feature map to at least one classifier based on hardware implementation for classification processing and obtains the classification result, it is specifically used to: send the pooled feature map to the first classifier and the second classifier to obtain the classification result of whether there is a situation where the code is missed or the code is directly bagged without being scanned.

[0165] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 6b Each step in the method embodiment.

[0166] In an embodiment of the image processing device of the present application, a convolution operation is performed on at least one target image using a hardware-based convolution device to obtain at least one shared feature map; candidate frame areas are extracted from at least one shared feature map respectively; a pooler only performs pooling processing on the candidate frame areas of the shared feature map whose number of candidate frame areas is greater than a set threshold, and directly skips the shared feature maps whose number is less than or equal to the set threshold; and a software-hardware combined image processing architecture is provided, which uses limited hardware resources to quickly perform convolution and pooling operations on images, thereby improving the efficiency of image processing.

[0167] Above Figure 7 The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0168] Above Figure 7 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0169] Above Figure 7 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0170] Above Figure 7The audio component in can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0171] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0176] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0177] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0178] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0179] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Inputting at least one target image into a hardware-based convolutional device; Performing feature extraction on at least one target image to obtain at least one shared feature map; Extracting candidate frame areas respectively from at least one shared feature map obtained by the convolver in real time through a compiler based on software implementation, and providing at least one shared feature map and the number of candidate frame areas thereof to a pooler based on hardware implementation; A target shared feature map is identified from at least one shared feature map in which the number of candidate frame areas is greater than a set threshold, and in the pooler, the candidate frame areas of the target shared feature map are pooled to obtain a pooled feature map.

2. The method according to claim 1, characterized in that Also includes: The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing.

3. The method according to claim 1, characterized in that Before inputting at least one target image into a hardware-based convolution device to extract features from at least one target image, the method further includes: Collecting a video clip to be convolved, wherein the video clip includes a plurality of images; At least one target image containing a target object is selected from the multiple images according to the displacement of pixels in the multiple images.

4. The method according to claim 2, characterized in that: The at least one classifier includes: a first classifier and a second classifier; wherein the first classifier is used to detect whether there is a situation where a code is not scanned in the target image; the second classifier is used to detect whether there is a situation where the target image is directly bagged without scanning the code; and further includes: If the classification result shows that the code is missed or the goods are directly packed without scanning, a warning operation will be performed.

5. The method according to claim 1, characterized in that Extract candidate box areas for at least one shared feature map, including: For the first shared feature map, a region generation algorithm is used to determine a plurality of candidate regions to be determined from the first shared feature map, wherein the first shared feature map is any one of the at least one shared feature map; Taking an area in the multiple candidate areas to be determined that is greater than or equal to a set probability threshold as a candidate frame area of ​​the first shared feature map; Count the number of candidate box areas of the first shared feature map.

6. The method according to any one of claims 1 to 5, characterized in that: Provide at least one shared feature map and the number of candidate box regions thereof to a hardware-based pooler, including: The number of the at least one shared feature map and its candidate box areas is written into a register corresponding to the pooler.

7. The method according to any one of claims 1 to 5, characterized in that: Pooling the candidate frame area of ​​the target shared feature map includes: Determine whether the size of the target shared feature map is greater than the feature map size threshold supported by the pooler; If so, the target shared feature map is split into multiple feature map segments, and the size of each feature map segment is smaller than the feature map size threshold; The multiple feature image segments are pooled to obtain a pooling result of the target shared feature map.

8. The method according to claim 4, characterized in that Collect the video clips to be convolved, including: Using a video sensor on an image processing device to collect video clips to be convolved; The image processing device is also provided with an indicator light and a speaker, and the warning operation includes at least one of the following: Control the indicator light on the image processing device to emit a light signal; The speaker on the image processing device is controlled to play a warning sound signal.

9. An image processing device, characterized in that: include: Processors, memory, and hardware-based convolution and pooling; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: Inputting at least one target image into a hardware-based convolutional device; Performing feature extraction on at least one target image to obtain at least one shared feature map; Extracting candidate frame areas respectively from at least one shared feature map obtained by the convolver in real time through a compiler based on software implementation, and providing at least one shared feature map and the number of candidate frame areas thereof to a pooler based on hardware implementation; A target shared feature map is identified from at least one shared feature map in which the number of candidate frame areas is greater than a set threshold, and the pooler performs pooling processing on the candidate frame areas of the target shared feature map to obtain a pooled feature map.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by one or more processors, the one or more processors are caused to perform the following actions: Inputting at least one target image into a hardware-based convolution device to extract features of the at least one target image, thereby obtaining at least one shared feature map; Extracting candidate frame areas respectively from at least one shared feature map obtained by the convolver in real time through a compiler based on software implementation, and providing at least one shared feature map and the number of candidate frame areas thereof to a pooler based on hardware implementation; A target shared feature map is identified from at least one shared feature map in which the number of candidate frame areas is greater than a set threshold, and in the pooler, the candidate frame areas of the target shared feature map are pooled to obtain a pooled feature map.

11. An image processing method, characterized in that: include: Acquire a video clip containing a code scanning action, wherein the video clip includes multiple images; Selecting at least one target image containing a person and a product from the plurality of images; Inputting at least one target image into a hardware-based convolutional device; Performing feature extraction on at least one target image to obtain at least one shared feature map; Extracting candidate frame areas respectively from at least one shared feature map obtained by the convolver in real time through a compiler based on software implementation, and providing at least one shared feature map and the number of candidate frame areas thereof to a pooler based on hardware implementation; Identifying a target shared feature map in which the number of candidate frame areas is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map; The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

12. The method according to claim 11, characterized in that The at least one classifier includes: a first classifier and a second classifier; wherein the first classifier is used to detect whether there is a situation where a code is missed in the target image; the second classifier is used to detect whether there is a situation where the code is directly bagged without being scanned in the target image, and the pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result, including: The pooled feature map is sent to the first classifier and the second classifier to obtain classification results of whether there is a situation where the code is missed or the code is directly bagged without being scanned.

13. An image processing device, characterized in that: include: Processors, memory, and hardware-based convolution and pooling; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: Acquire a video clip containing a code scanning action, wherein the video clip includes multiple images; Selecting at least one target image containing a person and a product from the plurality of images; Inputting at least one target image into a hardware-based convolutional device; Performing feature extraction on at least one target image to obtain at least one shared feature map; Extracting candidate frame areas respectively from at least one shared feature map obtained by the convolver in real time through a compiler based on software implementation, and providing at least one shared feature map and the number of candidate frame areas thereof to a pooler based on hardware implementation; Identifying a target shared feature map in which the number of candidate frame areas is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map; The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by one or more processors, the one or more processors are caused to perform the following actions: Acquire a video clip containing a code scanning action, wherein the video clip includes multiple images; Selecting at least one target image containing a person and a product from the plurality of images; Inputting at least one target image into a hardware-based convolutional device; Performing feature extraction on at least one target image to obtain at least one shared feature map; Extracting candidate frame areas respectively from at least one shared feature map obtained by the convolver in real time through a compiler based on software implementation, and providing at least one shared feature map and the number of candidate frame areas thereof to a pooler based on hardware implementation; Identifying a target shared feature map in which the number of candidate frame areas is greater than a set threshold from at least one shared feature map, and performing pooling processing on the candidate frame areas of the target shared feature map in the pooler to obtain a pooled feature map; The pooled feature map is sent to at least one classifier based on hardware implementation for classification processing to obtain a classification result.

Images