Image processing method, device, equipment, storage medium and computer program product

Abstract

The present application proposes an image processing method, device, equipment, storage medium and computer program product, which can be applied to various fields or scenarios such as cloud technology, artificial intelligence technology, object category recognition, object display recognition, etc., wherein the method includes: extracting features from the image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes the object to be recognized; obtaining a regional feature map corresponding to the object to be recognized based on the initial feature map; splitting the regional feature map into M sub-feature maps, and calling a feature fusion model to perform feature fusion processing on the M sub-feature maps to obtain fusion features; determining the recognition result of the object to be recognized based on the fusion features. By adopting the embodiments of the present application, on the one hand, automatic recognition of objects can be realized, and the efficiency of object recognition can be improved; on the other hand, the recognition accuracy of occluded objects can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, a computer device, a computer storage medium, and a computer program product. Background Art

[0002] For to B (i.e. to business, i.e., business startups are oriented to businesses and provide services to businesses), it is usually necessary to check whether their offline stores place goods as required, such as checking whether the specified goods are placed on shelves such as vending machines, self-service vending machines, and product display cabinets, and whether the goods on the shelves are displayed in accordance with the standards, etc. Since store inspections are costly and time-consuming, online inspections are usually adopted, that is, offline store staff take photos of the shelves, and then send the shelf photos to the background, and then the online inspection staff manually identify the goods in the uploaded photos to determine whether the offline stores place the goods as required. However, manual identification also has the problems of low efficiency and long time consumption. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, computer device and storage medium, which can realize automatic recognition of objects and improve the efficiency of object recognition on the one hand; on the other hand, they can improve the recognition accuracy of obscured objects.

[0004] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Performing feature extraction on the image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes an object to be identified;

[0006] Acquire a regional feature map corresponding to the object to be identified based on the initial feature map;

[0007] The regional feature map is split into M sub-feature maps, and a feature fusion model is called to perform feature fusion processing on the M sub-feature maps to obtain fusion features; M is a positive integer greater than 1;

[0008] Determining a recognition result of the object to be recognized based on the fusion feature;

[0009] Among them, the feature fusion model includes a feature fusion network, and the feature fusion network includes a first subnetwork and a second subnetwork connected in parallel, the first subnetwork corresponds to a first weight and a first activation function, and the second subnetwork corresponds to a second weight and a second activation function; the first subnetwork is used to process the input data based on the first weight and the first activation function to obtain an activation result, and the second subnetwork is used to process the input data based on the second weight and the second activation function to obtain an activation result; the output result of the feature fusion network is a fusion result determined based on the activation results output by the first subnetwork and the second subnetwork, and the fusion feature is determined based on the output result of the feature fusion network.

[0010] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0011] An extraction module, used for performing feature extraction on an image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes an object to be identified;

[0012] An acquisition module, configured to acquire a regional feature map corresponding to the object to be identified based on the initial feature map;

[0013] A fusion module is used to split the regional feature map into M sub-feature maps, and call a feature fusion model to perform feature fusion processing on the M sub-feature maps to obtain fusion features, where M is a positive integer greater than 1;

[0014] A determination module is used to determine the recognition result of the object to be recognized based on the fusion feature.

[0015] In one possible implementation,

[0016] The determination module is further used to call the first sub-network and the second sub-network of the first feature fusion network to process the first sub-feature map and the second sub-feature map respectively, and determine the first fusion result based on the activation results corresponding to the first sub-feature map and the second sub-feature map obtained by processing; the first sub-feature map and the second sub-feature map are any two different sub-feature maps among the M sub-feature maps;

[0017] The determination module is further used to call the first sub-network and the second sub-network of the second feature fusion network to process the third sub-feature map and the fourth sub-feature map respectively, and determine the second fusion result based on the activation results corresponding to the third sub-feature map and the fourth sub-feature map obtained by processing; the third sub-feature map and the fourth sub-feature map are any two different sub-feature maps among the M sub-feature maps;

[0018] The fusion module is further used to input the first fusion result and the second fusion result into the third feature fusion network for processing to obtain an intermediate fusion result;

[0019] The determination module is further used to determine the fusion features of the M sub-feature maps based on the intermediate fusion result.

[0020] In one possible implementation,

[0021] The determination module is further configured to, when the first sub-feature graph and the second sub-feature graph processed by the first feature fusion network are the same as the third sub-feature graph and the fourth sub-feature graph processed by the second feature fusion network, input the same sub-feature graph into different sub-networks for processing by the first feature fusion network and the second feature fusion network.

[0022] In one possible implementation,

[0023] The determination module is further used to call a sub-network of the fourth feature fusion network to process the intermediate fusion result to obtain an activation result corresponding to the intermediate fusion result;

[0024] The determination module is further used to call another sub-network of the fourth feature fusion network to process the regional feature map to obtain an activation result corresponding to the regional feature map;

[0025] The determination module is also used to determine a target fusion result based on the intermediate fusion result and the activation results respectively corresponding to the regional feature maps, and determine the fusion features of the M sub-feature maps based on the target fusion result.

[0026] In one possible implementation,

[0027] The determination module is further used to call the Kth convolution processing subnetwork of the feature extraction network to perform convolution processing on the image to be processed or the intermediate feature map of the image to be processed to obtain a processed intermediate feature map; K is any one of the set values, and the set value is part or all of the positive integers from 1 to X-1;

[0028] The determining module is further used to determine the first part of features and the second part of features from the processed intermediate feature map;

[0029] The extraction module is further used to output the first part of features to the K+1th convolution processing subnetwork of the feature extraction network for convolution processing, and output the second part of features to the feature splicing subnetwork;

[0030] The determination module is also used to call the feature splicing subnetwork to determine the initial feature map of the image to be processed based on the second part of the features of the Kth convolution processing subnetwork and the output result of the Xth convolution processing subnetwork.

[0031] In one possible implementation,

[0032] The determination module is further used to determine the reference area corresponding to the object to be identified in the initial feature map corresponding to each of the feature extraction networks;

[0033] The extraction module is further used to determine a target area based on the determined multiple reference areas, and to obtain a regional feature map corresponding to the object to be identified based on the target area and the initial feature maps corresponding to each of the feature extraction networks.

[0034] In one possible implementation,

[0035] The determination module is further used to determine the placement of the to-be-identified object in the to-be-processed image based on the identified object category and placement position of each to-be-identified object.

[0036] On the other hand, an embodiment of the present application provides a computer device, which includes: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program codes, and the processor is used to call program codes to execute the image processing method in the embodiment of the present application.

[0037] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the image processing method in the embodiment of the present application is executed.

[0038] Accordingly, an embodiment of the present application further provides a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the steps of the image processing method provided in the embodiment of the present application are implemented.

[0039] Accordingly, an embodiment of the present application also provides a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method provided in the embodiment of the present application.

[0040] The embodiment of the present application first extracts the initial feature map of the image to be processed, then obtains the regional feature map of the object to be identified based on the initial feature map, and then divides the regional feature map into regions to obtain multiple sub-feature maps. The multiple sub-feature maps are then feature-fused to obtain fusion features corresponding to the regional feature map, and finally the object to be identified is identified based on the fusion features to obtain a recognition result. On the one hand, automatic object recognition is achieved, thereby improving the efficiency of object recognition; on the other hand, because the regional feature map is divided into regions, and the divided regions are feature-fused with weights, the recognition accuracy of occluded objects can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1a is a schematic diagram of an image to be processed provided by an embodiment of the present application;

[0043] Figure 1b is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0044] Figure 2 is a flowchart of an image processing method provided in an embodiment of the present application;

[0045] Figure 3 The network structures of two feature extraction networks provided in the embodiments of the present application are shown;

[0046] Figure 4 Shows Figure 3 The two feature extraction networks in the figure correspond to the structures of the feature extraction models respectively;

[0047] Figure 5 The network structure for extracting the regional feature map of the object to be identified provided by the embodiment of the present application is shown;

[0048] Figure 6 The method of splitting the regional feature map and fusing the split feature maps is shown;

[0049] Figure 7 It is a flowchart of a duplicate checking processing method provided in an embodiment of the present application;

[0050] Figure 8 It shows how to perform dimensionality reduction on feature maps;

[0051] Fig. 9 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0052] Fig.10 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The embodiment of the present application provides an image processing method to realize automatic recognition of objects and improve the efficiency of object recognition. The image processing method provided in the embodiment of the present application can be implemented based on artificial intelligence (AI) technology. Among them, AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which mainly produces a new intelligent machine that can respond in a similar way to human intelligence by understanding the essence of intelligence, so that the intelligent machine has multiple functions such as perception, reasoning and decision-making.

[0055] AI technology is a comprehensive subject, which mainly includes several major directions such as machine learning (ML), computer vision technology (CV), speech processing technology, natural language processing technology and deep learning. The solution provided in the embodiment of this application involves technologies such as computer vision and machine learning under artificial intelligence technology. The following will describe computer vision technology and machine learning technology.

[0056] Computer Vision (CV) in AI technology is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR) technology, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional animation (3D) technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0057] Machine Learning (ML) is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Machine learning specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. This application mainly relates to artificial neural networks in machine learning technology. Specifically, the terminal device automatically extracts features from the collected image data and video data through artificial neural networks, automatically identifies the liveness detection results of the detection object, makes liveness detection more intelligent, and improves the accuracy of liveness detection.

[0058] In a feasible embodiment, the image processing method provided in the embodiment of the present application can also be implemented based on cloud technology and / or blockchain technology. Specifically, it may involve one or more of cloud storage, cloud database, and big data in cloud technology. For example, the data required to execute the image processing method (such as image data, etc.) is obtained from the cloud database. For another example, the data required to execute the image processing method can be stored in the form of blocks on the blockchain; the data generated by executing the image processing method (such as initial feature maps, regional feature maps, sub-feature maps, etc.) can be stored in the form of blocks on the blockchain; in addition, the data processing device that executes the image processing method can be a node device in the blockchain network.

[0059] Object category recognition technology is a technology based on image recognition technology that can be applied to offline merchants to inspect whether the display of goods is qualified. Object category recognition technology can effectively save labor costs and time costs, especially when applied to offline merchants. For merchants with a large number of offline stores, compared with manual visual verification, object category recognition technology can greatly reduce the verification cycle of the display, thereby making the verification results more real-time. In addition, object category recognition technology can also improve the efficiency and accuracy of verification, allowing stores to more conveniently and quickly understand the display dynamics of offline store goods.

[0060] When this application is used to inspect merchandise displays, the terminal device acquires image data of the object to be identified (for example, pictures of the store displays taken by the camera of an employee's mobile phone, pictures of the store displays taken by the store's real-time monitoring camera, etc.), and performs display identification based on the acquired image data to obtain an identification result. Based on the identification result, it is then determined whether the display of objects in the store meets the store's requirements (for example, whether the merchandise is placed in the corresponding location, whether promotional images of recent events are displayed, etc.). This effectively saves time in checking the display of store objects. For example Figure 1a As shown in Figure 1a, a shelf display picture taken and uploaded by a store clerk is used to identify the shelf display picture and obtain the category of items placed on the shelf (such as Figure 1a The application can identify objects on the shelf that are blocked and small in size when used to check the display of goods, thereby ensuring the accuracy of the object display recognition and more accurately determining whether the objects are placed correctly.

[0061] When the present application is applied to a road condition analysis scenario, the terminal device obtains image data of the object to be identified (e.g., a picture of the road condition near the vehicle taken by a vehicle-mounted camera, etc.), identifies the objects contained in the road condition based on the acquired picture to obtain an identification result (e.g., nearby pedestrians, obstacles on the road ahead, etc.), and analyzes the road condition based on the identification result. When applied to an in-vehicle Internet scenario, the present application can also identify objects that are obscured on the road condition and small objects, thereby ensuring the accuracy of the road condition analysis and improving the safety factor of driving.

[0062] It should be noted that the above-mentioned relevant data, such as the pictures of in-store displays taken by the cameras of the employees’ mobile phones, the pictures of in-store displays taken by the real-time monitoring cameras in the store, and the pictures of road conditions near the vehicle taken by the vehicle-mounted cameras, are all obtained with the permission or consent of the relevant owners.

[0063] See also Figure 1b , Figure 1b 1 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application. The image processing system mainly includes an image processing apparatus 101 and a terminal device 102.

[0064] The terminal device 102 is a device where the client of the image processing function platform is located, and is a device with communication functions, including but not limited to: smart phones, tablet computers, laptops, smart home appliances, vehicle-mounted terminals and other devices. The image processing device 101 includes an image processing chip, which can recognize and process images. Among them, the image processing device 101 can be embedded in the terminal device 102, or it can be an independent device different from the terminal device 102, which is not limited in the embodiments of the present application. When the image processing device 101 is an independent device, the image processing device 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0065] It should be noted that Figure 1b In the scenario shown, the number of terminal devices 102 and image processing devices 101 may be one or more, and this application does not impose any limitation thereto.

[0066] Optionally, when the image processing device 101 and the terminal device 102 are two independent devices, the operating object can select an image to be identified (such as a picture taken by the operating object, a picture in the operating object's photo album, etc.) by operating the terminal device 102, and send the image to the image processing device 101. After receiving the picture sent by the operating object, the image processing device 101 performs feature extraction on the picture to obtain an initial feature map corresponding to the picture. Then the image processing device 101 obtains a regional feature map corresponding to the object to be identified in the picture based on the initial feature map. The image processing device 101 then splits the regional feature map into multiple sub-feature maps, and fuses multiple sub-features corresponding to the multiple sub-feature maps to obtain a fused feature corresponding to the regional feature map. Finally, the image processing device 101 determines the recognition result corresponding to the object to be identified contained in the picture based on the fused feature.

[0067] The following will be combined Figures 2 to 8 The image processing method provided in the embodiment of the present application is described in detail.

[0068] See also Figure 2 , Figure 2 The image processing method provided in the embodiment of the present application is a flowchart. The image processing method includes steps S201 to S204.

[0069] S201, extracting features from an image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes an object to be identified.

[0070] The image to be processed is the image data to be processed, the initial feature map is the feature corresponding to the image to be processed, and the object to be identified is the object to be identified in the image to be processed. Figure 1a As shown, 1a may be an image to be processed, a beverage in 1a may be an object to be identified, and a price tag in 1a may be another object to be identified.

[0071] Optionally, each image is a matrix composed of pixel values, which may contain a variety of image information. The image information here may include the inherent parameter information of the image (such as image exposure, image grayscale, image size, etc.). Image features are extracted based on these inherent parameter information (for example, image feature extraction is performed on the image parameter information through detection operators), and image features of the image (such as image edge features, image texture features, image spatial relationship features, etc.) can be obtained. The image information here may also include feature information used to identify image features (such as features of products in pictures, etc.). Image features are extracted based on these feature information (for example, image feature extraction is performed on the feature information through convolutional neural networks), and image features of the image can be obtained.

[0072] Based on the above content, it can be concluded that there are two types of image feature extraction methods: one is a feature extraction method based on prior knowledge, and the detection operator of this feature extraction method is generally designed in advance and is obtained through a large amount of prior knowledge summary; the other is a feature extraction method based on deep learning, which includes convolutional neural networks (CNN). CNN feature extraction is equivalent to training one filter (filter, convolution kernel) after another. These filters are like the detection operators in the feature extraction method based on prior knowledge. In other words, CNN feature extraction is obtained by autonomous learning of neural networks. This embodiment mainly adopts a feature extraction method based on deep learning to obtain the initial feature map corresponding to the image to be processed and the feature information of the object to be identified, so as to provide a data basis for the subsequent acquisition of the regional feature map corresponding to the identified object based on the initial feature map corresponding to the processed image, so as to facilitate subsequent image processing operations. It should be noted that this embodiment can also adopt a feature extraction method based on prior knowledge, which is not limited here.

[0073] The following is a detailed description of the feature extraction method based on deep learning: The feature extraction method based on deep learning can make the extracted features more accurate, which is conducive to the subsequent identification of the object to be identified based on the extracted features. The CNN based on the feature extraction method of deep learning mainly includes the following key structures: convolution layer. Among them, the convolution layer is composed of a group of convolution units (filters, convolution kernels), which can also be understood as filters. Each filter extracts a specific feature contained in the image to be processed, and different convolution units extract different features. For example, input an image of a face, use a convolution unit to extract the features of the eyes, use another convolution unit to extract the features of the mouth, etc. The extraction process in the convolution layer can be a single feature transfer (convolution unit → convolution unit → convolution unit → convolution unit), or it can be to split the feature map output by some convolution units into two parts, one part flows to the next convolution unit, and the other part jumps to the bottom convolution unit.

[0074] In a possible embodiment, the initial feature map of the image to be processed is obtained by calling a feature extraction network to extract features from the image to be processed or the intermediate feature map of the image to be processed, and the feature extraction network includes a feature splicing subnetwork and X convolution processing subnetworks connected in series, where X is a positive integer greater than 1; wherein the feature extraction network is called to extract features from the intermediate feature map of the image to be processed or the image to be processed to obtain the initial feature map of the image to be processed, specifically: firstly calling the Kth convolution processing subnetwork of the feature extraction network to perform convolution processing on the image to be processed or the intermediate feature map of the image to be processed to obtain the processed intermediate feature map; K is any one of the set values, and the set value is part or all of the positive integers from 1 to X-1; then determining the first part of the features and the second part of the features from the processed intermediate feature map; then outputting the first part of the features to the K+1th convolution processing subnetwork of the feature extraction network for convolution processing, and outputting the second part of the features to the feature splicing subnetwork; finally, calling the feature splicing subnetwork to determine the initial feature map of the image to be processed based on the second part of the features of the Kth convolution processing subnetwork and the output result of the Xth convolution processing subnetwork. Optionally, the convolution processing subnetwork is a convolution unit in the convolution layer included in the CNN mentioned above. It can be understood that the convolution processing subnetwork is used to extract certain features contained in the image to be processed.

[0075] Optional, such as Figure 3 As shown, the above feature extraction network can be used as follows Figure 3 The network structure shown on the left, that is, the transmission method between the convolution units (or convolution processing subnetwork) in the feature extraction network is single feature transmission (convolution unit → convolution unit → convolution unit → convolution unit).

[0076] Optional, such as Figure 3 As shown, the above feature extraction network can be used as follows Figure 3 The network structure shown on the right is an improvement of CrossStagePartial (CSP for short). Figure 4 As shown in the CrossStagePartial on the right side of the figure. After experimental verification, the improved CSP structure can reduce 40% of the calculation amount, thereby achieving more efficient extraction of image features.

[0077] Optionally, the intermediate feature map of the image to be processed is the output of a certain layer of convolution processing subnetwork (for example, the feature map output by the third layer of convolution processing subnetwork is an intermediate feature map). The intermediate feature map contains one or more features of the image to be processed (for example, the intermediate feature map output by the third layer of convolution processing subnetwork contains the intermediate feature map extracted by the first layer of convolution processing subnetwork, the intermediate feature map extracted by the second layer of convolution processing subnetwork, and the intermediate feature map extracted by the third layer of convolution processing subnetwork). The first part of the features and the second part of the features are parts of the intermediate feature map output by a non-bottom layer convolution processing subnetwork (non-Xth layer of convolution processing subnetwork).

[0078] Optionally, the feature concatenation subnetwork is used to fuse the second part of the features of each convolution processing subnetwork and the output results of the underlying convolution processing subnetwork. The feature concatenation subnetwork can be a fully connected layer in CNN, which is used to integrate the features extracted by the convolution unit and can have local information with category distinction in the entire convolution layer.

[0079] Exemplarily, the feature extraction network includes 4 layers of convolution processing subnetworks, which are respectively: convolution processing subnetwork 1, convolution processing subnetwork 2, convolution processing subnetwork 3 and convolution processing subnetwork 4 (bottom-layer convolution processing subnetwork). After an image is input into the feature extraction network, convolution processing subnetwork 1 first performs feature extraction on the image to obtain an intermediate feature map 1, and then splits the intermediate feature map 1 into two parts: the first part feature 1 and the second part feature 1. Then the first part feature 1 is output to convolution processing subnetwork 2, and the second part feature 1 is output to convolution processing subnetwork 4. Convolution processing subnetwork 2 takes the first part feature 1 as input and performs feature extraction to obtain an intermediate feature. Figure 2 , the intermediate feature Figure 2 It is split into two parts: the first part feature 2 and the second part feature 2. Then the first part feature 2 is output to the convolution processing sub-network 3, and the second part feature 2 is output to the convolution processing sub-network 4. The convolution processing sub-network 3 takes the first part feature 2 as input and performs feature extraction to obtain the intermediate feature Figure 3 , the intermediate feature Figure 3 Output to the convolution processing subnetwork 4. It should be noted that the intermediate features Figure 3 The intermediate feature map 1 and / or the intermediate feature map 2 may be split and then input into the convolution processing sub-network 4, or may be directly input into the convolution processing sub-network 4 without splitting. Figure 2 It can also be directly input into the next convolution processing sub-network without splitting.

[0080] Optionally, when the extraction process of the convolution layer is a single feature transfer, the intermediate feature map output by each convolution processing sub-network is output to the next convolution processing sub-network, such as Figure 4As shown in the flowchart on the left, the feature extraction network is passed layer by layer. Exemplarily, the feature extraction network includes 4 layers of convolution processing subnetworks, which are: convolution processing subnetwork 1, convolution processing subnetwork 2, convolution processing subnetwork 3 and convolution processing subnetwork 4 (bottom convolution processing subnetwork). After an image is input into the feature extraction network, convolution processing subnetwork 1 first extracts features from the image to obtain an intermediate feature map 1, and outputs the intermediate feature map to convolution processing subnetwork 2. Then convolution processing subnetwork 2 extracts features from the intermediate feature map 1 to obtain an intermediate feature map. Figure 2 , and output the intermediate feature map to the convolution processing subnetwork 3. Then the convolution processing subnetwork 3 processes the intermediate feature map. Figure 2 Perform feature extraction to obtain intermediate features Figure 3 , and output the intermediate feature map to the convolution processing subnetwork 4. Finally, the convolution processing subnetwork 4 processes the intermediate feature Figure 3 The final convolutional layer extracts the feature map result.

[0081] Based on the above content, it can be concluded that there are two methods for extracting convolutional layers: one is the layer-by-layer progressive transfer extraction method; the other is the extraction method of splitting the output and then transferring it in a shunt. Compared with the layer-by-layer progressive transfer extraction method, the extraction method of splitting the output and then transferring it in a shunt can reduce computing energy consumption and computing time without affecting recognition accuracy.

[0082] S202: Obtain a regional feature map corresponding to the object to be identified based on the initial feature map.

[0083] Among them, the regional feature map corresponding to the object to be identified can be an image included in the candidate box generated based on the object to be identified. The initial feature map is the output of the convolution layer. Based on the initial feature map, the regional feature map corresponding to the object to be identified is obtained: based on the output of the convolution layer, regional feature maps are generated for the objects to be identified. In other words, different convolution processing subnetworks (convolution units) in the convolution layer are used to extract different features from the image to be processed, and the output of each convolution processing subnetwork corresponds to a feature of the image to be processed, and multiple convolution processing subnetworks extract multiple features of the image to be processed. Then, candidate boxes corresponding to the object to be identified are generated based on these features. For example, Figure 1a As shown, a certain convolution processing sub-network is used to extract features of the image shown in 1a (in this case, 1a is the image to be processed), and the features of the water bottle (the object to be identified) located on the far left are obtained. According to the features of the water bottle, a candidate frame (regional feature map) is generated for the water bottle; another convolution processing sub-network is used to extract features of the image shown in 1a again, and the features of the promotional stand (the object to be identified) are obtained. According to the features of the promotional stand, a candidate frame (regional feature map) is generated for the promotional stand.

[0084] In a possible embodiment, a feature extraction network is included in a feature extraction model, and the feature extraction model includes a plurality of feature extraction networks connected in series, the input data of the Yth feature extraction network is the output result of the Y-1th feature extraction network, Y is a positive integer greater than 1, and each feature extraction network obtains an initial feature map of the image to be processed after processing the input data, and the feature scales of the initial feature maps corresponding to the respective feature extraction networks are different; wherein, a regional feature map corresponding to the object to be identified is obtained based on the initial feature map, specifically as follows: first, a reference area corresponding to the object to be identified in the initial feature map corresponding to each feature extraction network is determined; then, a target area is determined based on the determined plurality of reference areas, and based on the target area and the initial feature maps corresponding to each feature extraction network, a regional feature map corresponding to the object to be identified is obtained.

[0085] Optionally, the feature scale can be expressed as a receptive field. The receptive field of a deep network is relatively large, but the resolution of the feature map is low. On the contrary, the receptive field of a low-level network is small, but the resolution of the feature map is high. The receptive field can be understood as the field of vision of the human eye. When the distance to the object is far, the whole object can be seen, but the details of the object cannot be seen clearly; when the distance to the object is close, only part of the object can be seen, but the details of the part can be seen clearly. The feature scale can also be expressed as dimension, description details, etc. Similarly, the description details of a deep network are poor, and the description details of a relatively low-level network are better.

[0086] Optionally, the reference area is an area related to the object to be identified that corresponds to features extracted by a multi-layer convolution processing subnetwork in the feature extraction network.

[0087] Optionally, the regional feature map is obtained based on the initial feature map, and the feature scales of the initial feature maps corresponding to each feature extraction network are different. Based on the above explanation of the feature scale, it can be concluded that the feature map with richer details has a smaller extracted area (i.e., a smaller receptive field), and the feature map with less details may have a larger extracted area (i.e., a larger receptive field), thereby determining the regional feature map based on different feature scales can more accurately obtain the regional feature map.

[0088] In a possible embodiment, based on the outputs of multiple feature extraction networks, an initial feature map related to the object to be identified is determined from all initial feature maps, and multiple regions related to the object to be identified in the image to be processed are determined based on the multiple related initial feature maps. The multiple regions are multiple reference regions corresponding to the object to be identified. For example, the image to be processed contains multiple objects: a table, a stool, and a kitten. The object to be identified is a kitten, and the initial feature maps corresponding to three feature extraction networks are the kitten's head, the kitten's body, and the kitten's tail. The three initial feature maps corresponding to the three feature extraction networks are selected, and based on the positions of the three initial feature maps in the image to be processed, the region related to the kitten, that is, the reference region corresponding to the kitten, is determined.

[0089] In a possible embodiment, determining the target area based on multiple reference areas can be to select a reference area from multiple reference areas as the target area, and then determine the regional feature map corresponding to the target area from all initial feature maps according to the position of the target area in the image to be processed. Exemplarily, the image to be processed contains multiple objects: a table, a stool, and a kitten. The object to be identified is a kitten, and there are three reference areas: reference area 1: only contains the whole kitten, reference area 2: contains part of the kitten, and reference area 3: contains not only the whole kitten but also the table. From these three reference areas, select reference area 1 that only completely contains the kitten as the target area, and then determine the regional feature map corresponding to the target area from all initial feature maps according to the position of the target area in the image to be processed, that is, the regional feature map corresponding to a complete kitten.

[0090] Optionally, the regional feature map can also be obtained by fusing the feature maps of multiple reference areas. That is, the features of multiple reference areas containing the object to be identified in each initial feature map are fused to obtain a regional feature map containing the object to be identified. For example, the image to be processed contains multiple objects: a table, a stool, and a kitten. The object to be identified is a kitten, and there are three reference areas: reference area 1: containing the head of the kitten, reference area 2: containing the tail of the kitten, and reference area 3: not containing the tail and body of the kitten. The object to be identified (kitten) in the image to be processed is composed of these three areas. The feature corresponding to reference area 1 is reference area feature 1, the feature corresponding to reference area 2 is reference area feature 2, and the feature corresponding to reference area 3 is reference area feature 3. The reference area feature 1, the reference area feature 2, and the reference area feature 3 are fused and then predicted to obtain the regional feature map.

[0091] In order to better understand the above feature fusion, the feature fusion is further explained below. First of all, fusing features of different scales can improve segmentation performance, that is, efficiently fusing low-level features with higher resolution (i.e., containing more position and detail information) with high-level features with stronger semantic information (i.e., low resolution and poor detail perception). Feature fusion is divided into early fusion and late fusion according to the order of fusion and prediction. Among them, early fusion is to fuse multiple layers of features first, and then train the predictor on the fused features, that is, after complete fusion, unified detection is performed; late fusion is to start detection on the partially fused layer before the final fusion is completed, so there will be multiple layers of detection, and finally multiple detection results (feature maps) will be fused. Among them, there are two representative ideas for late fusion: the first is that features are not fused, multi-scale features are predicted separately, and then the prediction results are integrated, such as Single Shot MultiBox Detector (SSD) and Multiscale-Convolutional Neural Network (MS-CNN); the second is pyramid fusion of features, which is performed first and then predicted, such as Feature Pyramid Network (FPN).

[0092] Optionally, this embodiment combines the feature pyramid network FPN with the spatial pyramid pool (Spatial PyramidPooling, SPP) to fuse the features, and then predict the regional feature map corresponding to the object to be identified. Compared with other prediction algorithms (only using top-level features for prediction), the feature pyramid network FPN fuses the top-level features with the bottom-level features through upsampling. Because although the bottom-level feature information is small (the receptive field of the previous text and is small), there are more details (such as more accurate position), so the feature pyramid network FPN fuses the bottom-level features with the top-level features for prediction, which can make it possible to predict and identify the tiny objects and obscured objects contained in the image to be processed. The spatial pyramid pool SPP is a pooling layer, which is also called a downsampling layer (i.e., feature dimensionality reduction), which is mainly used to reduce the image and compress the data while ensuring that the image features remain unchanged. Compared with other pooling layers (directly cropping the image to a fixed size that CNN can handle), spatial pyramid pooling SPP can remove CNN's requirement for fixed input, that is, there is no need to crop the image to a fixed size that CNN can handle, and the complete image can be retained, making the prediction result more accurate.

[0093] Based on the above content, it can be concluded that the feature pyramid network FPN performs upsampling to obtain more details of the image, while the spatial pyramid pool SPP performs downsampling to remove the requirement of CNN for fixed input, so that the output of the feature pyramid network FPN can be input into CNN. Figure 5 As shown, Figure 5 The process marked by 501 is the upsampling process performed by the feature pyramid network FPN. Figure 5 The process marked by 502 is the downsampling process performed by the spatial pyramid pool SPP.

[0094] S203, splitting the regional feature map into M sub-feature maps, and calling the feature fusion model to perform feature fusion processing on the M sub-feature maps to obtain fusion features; M is a positive integer greater than 1.

[0095] Among them, the sub-feature map can be understood as splitting the regional feature map into multiple regions, and each region is a sub-feature map. Figure 6 The area marked with 603 is a regional feature map, which is divided into four areas (upper area, lower area, left area and right area), and these four areas are four sub-feature maps.

[0096] Among them, the feature fusion model includes a feature fusion network, the feature fusion network includes a first sub-network and a second sub-network connected in parallel, the first sub-network corresponds to a first weight and a first activation function, and the second sub-network corresponds to a second weight and a second activation function; the first sub-network is used to process the input data based on the first weight and the first activation function to obtain an activation result, and the second sub-network is used to process the input data based on the second weight and the second activation function to obtain an activation result; the output result of the feature fusion network is a fusion result determined based on the activation results output by the first sub-network and the second sub-network, and the fusion feature is determined based on the output result of the feature fusion network.

[0097] Optionally, the activation function is used to introduce nonlinear factors so that the neural network can arbitrarily implement any nonlinear function. The first weight and the second weight are two different weights, which can be preset manually or generated by a computer based on prior experience, and there is no limitation here. Figure 6 In the figure, the region marked with 601 is the internal structure of a feature fusion network. The feature fusion network includes two processes, which are two sub-networks. Figure 2 ,enter Figure 6 After the feature fusion network in the first sub-network, the first sub-network performs convolution on feature map 1 and then performs convolution again with weight 1, and then inputs the activation function to obtain the activation result 1; the second sub-network performs convolution on feature map 1 and performs convolution again with weight 1, and then inputs the activation function to obtain the activation result 1; Figure 2After the convolution, the weight 2 is added and the convolution is performed again, and then the activation function is input to obtain the activation result 2, and finally the output result is obtained according to the activation result 1 and the activation result 2. It should be noted that the two activation functions included in the feature fusion network can be the same activation function (for example, both activation functions are Mish functions), or the two activation functions can be different activation functions (for example, one activation function is Mish and the other activation function is Tanh function), and there is no restriction here.

[0098] In a possible embodiment, the feature fusion model includes multiple feature fusion networks, the multiple feature fusion networks include a first feature fusion network, a second feature fusion network and a third feature fusion network, the first feature fusion network and the second feature fusion network are connected in parallel, and the input end of the third feature fusion network is connected to the output end of the first feature fusion network and the second feature fusion network; wherein, calling the feature fusion model to perform feature fusion processing on the M sub-feature graphs to obtain fused features is specifically as follows: firstly, calling the first sub-network and the second sub-network of the first feature fusion network to process the first sub-feature graph and the second sub-feature graph respectively, and based on the activations corresponding to the processed first sub-feature graph and the second sub-feature graph respectively The result is used to determine the first fusion result; then the first sub-feature map and the second sub-feature map are any two different sub-feature maps among the M sub-feature maps; then the first sub-network and the second sub-network of the second feature fusion network are called to process the third sub-feature map and the fourth sub-feature map respectively, and based on the activation results corresponding to the processed third sub-feature map and the fourth sub-feature map, the second fusion result is determined; then the third sub-feature map and the fourth sub-feature map are any two different sub-feature maps among the M sub-feature maps; finally, the first fusion result and the second fusion result are input into the third feature fusion network for processing to obtain an intermediate fusion result; based on the intermediate fusion result, the fusion features of the M sub-feature maps are determined.

[0099] The first sub-feature graph may be any one of the M sub-feature graphs, and the second sub-feature graph may be any one of the M sub-feature graphs except the first sub-feature graph. The feature fusion network may be used to determine the first fusion result based on the activation results corresponding to the first sub-feature graph and the second sub-feature graph obtained by processing. In other words, the first sub-feature graph and the second sub-feature graph are input into the feature fusion network together to obtain the first fusion result.

[0100] In order to better understand the feature fusion model, Figure 6 An example is given to further explain the feature fusion model. Figure 6The area marked with 602 is a feature fusion model, and the Region Assembly Block (RAB) in 602 is a feature fusion network. The feature fusion model 602 includes multiple RABs. The first feature fusion network is the two parallel RABs on the leftmost side of the feature fusion model 602. The first subnetwork and the second subnetwork included in the first feature fusion network correspond to the two parallel RABs respectively. The second feature fusion network can be understood as a RAB in the middle of the feature fusion model 602. The third feature fusion network can be understood as the rightmost RAB in the feature fusion model 602. It should be noted that the first feature fusion network can also have only one RAB, or more than two RABs; the second feature fusion network and the third feature fusion network can also have more than one RAB. For the convenience of subsequent description, Figure 6 The first feature fusion network has two parallel RABs, the second feature fusion network has one RAB, and the third feature fusion network has one RAB as an example for description.

[0101] Combine the following Figure 6 This step is further explained with an example. Figure 6 603 in the figure is a regional feature map, which is divided into four sub-feature maps: the upper part ( Figure 6 Partial top in 603), lower part ( Figure 6 Partial bottom in 603), left part ( Figure 6 Partial left) and right part ( Figure 6 The left part and the right part are input into the first sub-network (such as the upper RAB of the two leftmost RABs) of the first feature fusion network. Figure 6 The upper part of the route in 601) and the second sub-network (such as Figure 6 Similarly, the left part and the upper part are input into the second feature fusion network (the lower RAB of the two leftmost RABs) to obtain fusion result 2, the left part and the lower part are input into the second feature fusion network to obtain fusion result 3, the right part and the upper part are input into the first feature fusion network to obtain fusion result 4, the right part and the lower part are input into the second feature fusion network to obtain fusion result 5, the upper part and the lower part are input into the second feature fusion network to obtain fusion result 6. These six fusion results are input into the third feature fusion network ( Figure 6The RAB in the middle of 602 is obtained to obtain multiple intermediate fusion results. Then, the fusion features of the M sub-feature maps are determined according to the multiple intermediate fusion results. Thus, the occluded objects and minor problems can also be accurately identified, reducing the influence of the occluded area on the recognition result.

[0102] In a possible embodiment, when the first sub-feature map and the second sub-feature map processed by the first feature fusion network are the same as the third sub-feature map and the fourth sub-feature map processed by the second feature map fusion network, the first feature fusion network and the second feature fusion network input the same sub-feature map into different sub-networks for processing.

[0103] When the first sub-feature map and the second sub-feature map processed by the first feature fusion network are the same as the third sub-feature map and the fourth sub-feature map processed by the second feature fusion network, the first feature fusion network and the second feature fusion network allow the same sub-feature map to carry different weights for fusion processing.

[0104] For example, when the sub-feature graph 1 is first input into the first sub-network of the first feature fusion network, the sub-feature Figure 2 After the first input to the second sub-network of the first feature fusion network, when sub-feature graph 1 and sub-feature Figure 2 When the first feature fusion network is input again, sub-feature graph 1 is input into the second sub-network of the first feature fusion network, and sub-feature Figure 2 Input the first sub-network of the first feature fusion network. That is, sub-feature graph 1 carries weight 1 and sub-feature Figure 2 After carrying weight 2 for fusion, when sub-feature map 1 and sub-feature Figure 2 When fusion is performed again, sub-feature map 1 carries weight 2 and sub-feature Figure 2 Carry weight 1 for fusion. Similarly, for the third sub-feature map and the fourth sub-feature map input by the second feature fusion network, when they are input into the second feature fusion network for the second time, the weights carried by the two inputs are different, that is, the sub-networks of the two inputs are different.

[0105] Optionally, for the parallel first feature fusion network and the second feature fusion network, in order to save processing time and improve processing efficiency, the sub-feature graphs can be processed in the following manner: first, the sub-feature graphs are divided into groups of two. Then, the same group is simultaneously input into the first feature fusion network and the second feature fusion network according to the grouping situation. At this time, the first feature fusion network and the second feature fusion network are the same feature fusion network. When a sub-feature graph in the same group enters the two feature fusion networks, it is used as the input of the first sub-network when it is input into the first feature fusion network, and as the input of the second sub-network when it is input into the second feature fusion network; when another sub-feature graph in the same group enters the two feature fusion networks, it is used as the input of the second sub-network when it is input into the first feature fusion network, and as the input of the third sub-network when it is input into the second feature fusion network.

[0106] Exemplarily, there are four sub-feature graphs: sub-feature graph 1, sub-feature Figure 2 , sub-features Figure 3 And sub-features Figure 4 The four sub-feature graphs are divided into two groups, and the total number of groups is divided into five groups: sub-feature graph 1 and sub-feature Figure 2 Divided into a group, sub-feature Figure 1 and sub-feature Figure 3 Divided into a group, sub-feature Figure 1 and sub-feature Figure 4 Divide into a group, sub-features Figure 2 and sub-features Figure 3 Divide into a group, sub-features Figure 3 and sub-features Figure 4 Divide into a group. Based on sub-features Figure 1 and sub-features Figure 2 The group is input into the first feature fusion network and the second feature fusion network at the same time, wherein the sub-feature graph 1 is input into the first sub-network of the first feature fusion network and the second sub-network of the second feature fusion network respectively; Figure 2 The second sub-network of the first feature fusion network and the first sub-network of the second feature fusion network are respectively input. Then the fusion process is performed to obtain the intermediate fusion result. Similarly, the other four groups are also input into the first feature fusion network and the second feature fusion network in the same way for fusion processing. It should be noted that sub-feature graph 1 can also be input into the second sub-network of the first feature fusion network and the first sub-network of the second feature fusion network. As long as the same sub-feature graph is input into different sub-networks of the two feature fusion networks, the processing methods of the other four groups will not be repeated.

[0107] In a possible embodiment, the multiple feature fusion networks also include a fourth feature fusion network, and the input end of the fourth feature fusion network is connected to the output end of the third feature fusion network; the fusion features of the M sub-feature maps are determined based on the intermediate fusion result, specifically: first, a sub-network of the fourth feature fusion network is called to process the intermediate fusion result, and an activation result corresponding to the intermediate fusion result is obtained; another sub-network of the fourth feature fusion network is called to process the regional feature map, and an activation result corresponding to the regional feature map is obtained; then, based on the intermediate fusion result and the activation results corresponding to the regional feature map, a target fusion result is determined, and the fusion features of the M sub-feature maps are determined based on the target fusion result.

[0108] Among them, the fourth feature fusion network is used to fuse the intermediate fusion result with the regional feature map to obtain the fusion feature, so as to determine the positional relationship between the objects to be identified, thereby judging the placement of the objects to be identified.

[0109] In a possible embodiment, the feature fusion network may further include a feature fusion module (such as Figure 6 The feature fusion module fuses the two activation results. Figure 6 As shown, Figure 6 The feature fusion model marked with 602 includes 4 RABs (feature fusion networks), and each RAB includes a MAX (feature fusion module).

[0110] In a possible embodiment, the fusion method of the feature fusion module can be: select an activation result with a larger value from the two input activation results as the output. The two sub-feature maps are respectively input into the two sub-networks of the feature fusion network for processing. In the two sub-networks, the two sub-feature maps are multiplied by the corresponding weights. Then activation is performed to obtain two activation results corresponding to the two sub-feature maps, and the larger activation result is selected from the two activation results as the target fusion result. It should be noted that the target fusion result can be the fusion feature of M sub-feature maps, or the fusion feature of some sub-feature maps in the M sub-feature maps, or the fusion feature of M sub-feature maps and regional feature maps. Figure 6 As shown in the feature fusion network 602, the target fusion result output by the rightmost RAB is the fusion feature of M sub-feature maps and the regional feature map, the target fusion result output by the leftmost RAB is the fusion feature of some sub-feature maps in the M sub-feature maps, and the middle RAB is the fusion feature of M sub-feature maps. Exemplarily, the two sub-feature maps are: sub-feature map 1 and sub-feature map Figure 2; The two weights are weight 1 and weight 2. Sub-feature map 1 is input into the first sub-network of the feature fusion network for convolution processing, then multiplied by weight 1 and convolved again, and then input into the activation function to obtain activation result 1; ... Figure 2 After inputting into the second sub-network of the feature fusion network, convolution processing is performed, and then multiplied by weight 2 and convolution processing is performed again before inputting into the activation function to obtain activation result 2. Activation result 1 and activation result 2 are input into the feature fusion module, and the feature fusion module compares activation result 1 and activation result 2 and selects the activation result with a larger value as the output.

[0111] S204: Determine the recognition result of the object to be recognized based on the fused features.

[0112] The recognition result may include the object category of the object to be recognized (such as the object to be recognized is a beverage of xx brand), and may also include the position information of the object to be recognized (the position in the image to be processed), etc.

[0113] Optionally, the fused features are input into a feature classification network to obtain the matching probability between the object to be identified and multiple category labels, and then the identification result of the object to be identified is determined based on the matching probability. Among them, the multiple category labels can be set by the merchant, for example, the category label can be the product number set by the store, etc., or it can be the default of the model.

[0114] In a possible embodiment, in order to obtain the matching probability between the object to be identified and multiple identification labels, the fused features are output to the fully connected layer of the feature classification network, and then the binary classification probability vector of the preset scale is output through the normalized exponential function, and the category label is determined according to the selection rule (for example, selecting the category with the largest probability value). Exemplarily, the probability that the object to be identified output by the fused features through the feature classification network corresponds to the "beverage" label is 0.7, and the probability that the object to be identified corresponds to labels other than the "beverage" label is less than 0.7, then the object to be identified is judged to be a beverage. Among them, the normalized exponential function mentioned above is used to normalize the gradient logarithm of a finite discrete probability distribution. The normalized exponential function includes multinomial logistic regression and multinomial linear discriminant analysis.

[0115] In a possible embodiment, the position information of the object to be identified is obtained based on the image space relationship features contained in the regional feature map corresponding to the object to be identified, or based on the position relationship of the regional feature map corresponding to the object to be identified in the initial feature map corresponding to the image to be processed.

[0116] In a possible embodiment, there are one or more objects to be identified in the image to be processed, and the identification result of each object to be identified includes an object category and a placement position; therefore, a method for determining the placement status of the object may be: based on the identified object category and placement position of each object to be identified, determine the placement status of the object to be identified in the image to be processed.

[0117] Optionally, the image to be processed is obtained by photographing a shelf where the object to be identified is located. The placement condition includes whether the position of the object to be identified on the shelf is correct, whether the orientation of the object to be identified is correct, and the like.

[0118] For example, Figure 1a As shown, Figure 1a Take an image of a shelf (image to be processed) and take any product (such as the beverage in the picture) as an example, and take the product as the item to be identified. Through the above steps, it can be determined that the item to be identified is a bottle of xx brand beverage, and the bottle of xx brand beverage is placed on the right side of the shelf. According to the identification plate on the shelf, it is determined that the logo on the identification plate is also "xx brand beverage", and it is judged that the placement of the bottle of xx brand beverage is correct. And according to the regional feature map corresponding to the bottle of xx brand beverage, it can be determined that the placement direction of the bottle of xx brand beverage is the same as the preset expected placement direction, so it can be judged that the placement direction of the bottle of xx brand beverage is correct.

[0119] The embodiment of the present application also provides a duplicate checking algorithm for uploaded images to be processed. The duplicate checking algorithm matches the initial features corresponding to the uploaded image to be processed with the features in the image feature library to determine whether the uploaded image to be processed is an image that has been uploaded historically. Exemplarily, a merchant can create an image feature library through an interface, and the image feature library is used to store the images that the merchant has uploaded historically. When the merchant has newly uploaded images, it can initiate a duplicate checking application, submit a duplicate checking time period and the feature library that needs to be checked for duplicate checking. The main process of the duplicate checking algorithm is as follows: Figure 7 Among them, a merchant can create multiple image feature libraries corresponding to the merchant according to needs. For example, if a merchant has offline stores all over the country, the merchant can establish multiple image feature libraries according to the regions where the offline stores are located.

[0120] Optionally, in order to allow the image feature library to store more image features and to reduce the time spent on duplicate checking, before storing the uploaded image in the image feature library, feature extraction is performed on the image, and dimensionality reduction is performed on the extracted image features. The dimensionality reduction method used can be based on the Global Descriptor Table (GDT) combined with Principal Component Analysis (PCA). PCA is mainly used to extract the main feature components of the data, thereby achieving the purpose of dimensionality reduction. For example, the above feature extraction process can be Figure 8 As shown, Figure 8 GD-1 to GD-N represent global descriptors (GD), FC(1) to FC(n) are obtained by processing GD-1 to GD-N respectively using a fully connected network layer, and X-w1 to X-wn are obtained by processing FC(1) to FC(n) respectively using a residual network. For example, X-w1 is obtained by inputting FC(1) into Conv-1 for processing and then fusing it with FC(1). Figure 8 The leftmost image in the figure is the input image (i.e., the image to be processed), and multiple feature extractions (including downsampling, pooling, etc.) are performed on the input image to obtain multiple global feature description operators ( Figure 8 Then, the multiple global feature description operators are input into the corresponding fully connected network layer and the residual network for processing to obtain multiple output results ( Figure 8 The output features after dimensionality reduction can be obtained by merging multiple output results. Finally, these feature data are stored in the feature library as historical uploaded features for subsequent duplication checking.

[0121] See also Fig. 9 , Fig. 9 : is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device provided in an embodiment of the present application includes:

[0122] like Fig. 9901 is an extraction module 901, which is used to extract features of the image to be processed and obtain an initial feature map of the image to be processed, wherein the image to be processed includes the object to be identified; 902 is an acquisition module 902, which is used to obtain a regional feature map corresponding to the object to be identified based on the initial feature map; 903 is a fusion module 903, which is used to split the regional feature map into M sub-feature maps, and call a feature fusion model to perform feature fusion processing on the M sub-feature maps to obtain fusion features, where M is a positive integer greater than 1; 904 is a determination module 904, which is used to determine the recognition result of the object to be identified based on the fusion features; wherein the feature fusion model The model includes a feature fusion network, which includes a first sub-network and a second sub-network connected in parallel, the first sub-network corresponds to a first weight and a first activation function, and the second sub-network corresponds to a second weight and a second activation function; the first sub-network is used to process the input data based on the first weight and the first activation function to obtain an activation result, and the second sub-network is used to process the input data based on the second weight and the second activation function to obtain an activation result; the output result of the feature fusion network is a fusion result determined based on the activation results output by the first sub-network and the second sub-network, and the fusion feature is determined based on the output result of the feature fusion network.

[0123] In another implementation, the determination module 904 is further used to call the first sub-network and the second sub-network of the first feature fusion network to process the first sub-feature map and the second sub-feature map respectively, and determine the first fusion result based on the activation results corresponding to the first sub-feature map and the second sub-feature map obtained by processing; the first sub-feature map and the second sub-feature map are any two different sub-feature maps among the M sub-feature maps; the determination module 904 is further used to call the first sub-network and the second sub-network of the second feature fusion network to process the third sub-feature map and the fourth sub-feature map respectively, and determine the second fusion result based on the activation results corresponding to the third sub-feature map and the fourth sub-feature map obtained by processing; the third sub-feature map and the fourth sub-feature map are any two different sub-feature maps among the M sub-feature maps; the fusion module 903 is further used to input the first fusion result and the second fusion result into the third feature fusion network for processing to obtain an intermediate fusion result; the determination module 904 is further used to determine the fusion features of the M sub-feature maps based on the intermediate fusion result.

[0124] In another implementation, the determination module 904 is also used to, when the first sub-feature graph and the second sub-feature graph processed by the first feature fusion network are the same as the third sub-feature graph and the fourth sub-feature graph processed by the second feature fusion network, the first feature fusion network and the second feature fusion network input the same sub-feature graph into different sub-networks for processing.

[0125] In another implementation, the determination module 904 is further used to call a sub-network of the fourth feature fusion network to process the intermediate fusion result to obtain an activation result corresponding to the intermediate fusion result; the determination module 904 is further used to call another sub-network of the fourth feature fusion network to process the regional feature map to obtain the activation result corresponding to the regional feature map; the determination module 904 is further used to determine the target fusion result based on the intermediate fusion result and the activation results corresponding to the regional feature map, and determine the fusion features of the M sub-feature maps based on the target fusion result.

[0126] In another implementation, the determination module 904 is also used to call the Kth convolution processing subnetwork of the feature extraction network to perform convolution processing on the image to be processed or the intermediate feature map of the image to be processed to obtain the processed intermediate feature map; K is any one of the set values, and the set value is part or all of the positive integers from 1 to X-1; the determination module 904 is also used to determine the first part of the features and the second part of the features from the processed intermediate feature map; the extraction module 901 is also used to output the first part of the features to the K+1th convolution processing subnetwork of the feature extraction network for convolution processing, and output the second part of the features to the feature splicing subnetwork; the determination module 904 is also used to call the feature splicing subnetwork to determine the initial feature map of the image to be processed based on the second part of the features of the Kth convolution processing subnetwork and the output result of the Xth convolution processing subnetwork.

[0127] In another implementation, the determination module 904 is also used to determine the reference area corresponding to the object to be identified in the initial feature map corresponding to each feature extraction network; the extraction module 901 is also used to determine the target area based on the determined multiple reference areas, and obtain the regional feature map corresponding to the object to be identified based on the target area and the initial feature map corresponding to each feature extraction network.

[0128] In another implementation, the determination module 904 is further configured to determine the placement of the objects to be identified in the image to be processed based on the identified object categories and placement positions of the objects to be identified.

[0129] It can be understood that the functions of each functional unit of the image processing device provided in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment, and its specific implementation process can refer to the relevant description in the above method embodiment, which will not be repeated here.

[0130] In other feasible embodiments, the image processing device provided in the embodiment of the present application may also be implemented in a combination of software and hardware. As an example, the image processing device provided in the embodiment of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiment of the present application. For example, the processor in the form of a hardware decoding processor may adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0131] See also Fig.10 , is a structural diagram of a computer device provided in an embodiment of the present application, and the computer device 100 may include a processor 1001, a memory 1002, a network interface 1003 and at least one communication bus 1004. Among them, the processor 1001 is used to schedule computer programs, and may include a central processing unit, a controller, and a microprocessor; the memory 1002 is used to store computer programs, and may include a high-speed random access memory RAM, a non-volatile memory, such as a disk storage device, a flash memory device; the network interface 1003 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and provide data communication functions, and the communication bus 1004 is responsible for connecting various communication elements. The computer device 100 may correspond to the data processing device 100 in the foregoing text. The memory 1002 is used to store a computer program, and the computer program includes program instructions. The processor 1001 is used to execute the program instructions stored in the memory 1002 to execute the process described in steps S301 to S304 in the above embodiment, and perform the following operations:

[0132] In one implementation, the processor 1001 performs feature extraction on the image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes an object to be identified; based on the initial feature map, a regional feature map corresponding to the object to be identified is obtained; the regional feature map is split into M sub-feature maps, and a feature fusion model is called to perform feature fusion processing on the M sub-feature maps to obtain a fusion feature; M is a positive integer greater than 1; and a recognition result of the object to be identified is determined based on the fusion feature;

[0133] In one implementation, the processor 1001 calls the first sub-network and the second sub-network of the first feature fusion network to process the first sub-feature map and the second sub-feature map respectively, and determines the first fusion result based on the activation results corresponding to the first sub-feature map and the second sub-feature map obtained by processing; the first sub-feature map and the second sub-feature map are any two different sub-feature maps in the M sub-feature maps; calls the first sub-network and the second sub-network of the second feature fusion network to process the third sub-feature map and the fourth sub-feature map respectively, and determines the second fusion result based on the activation results corresponding to the third sub-feature map and the fourth sub-feature map obtained by processing; the third sub-feature map and the fourth sub-feature map are any two different sub-feature maps in the M sub-feature maps; the first fusion result and the second fusion result The input data is input into a third feature fusion network for processing to obtain an intermediate fusion result; based on the intermediate fusion result, fusion features of the M sub-feature graphs are determined; wherein the feature fusion model includes a feature fusion network, the feature fusion network includes a first sub-network and a second sub-network connected in parallel, the first sub-network corresponds to a first weight and a first activation function, and the second sub-network corresponds to a second weight and a second activation function; the first sub-network is used to process the input data based on the first weight and the first activation function to obtain an activation result, and the second sub-network is used to process the input data based on the second weight and the second activation function to obtain an activation result; the output result of the feature fusion network is a fusion result determined based on the activation results output by the first sub-network and the second sub-network, and the fusion feature is determined based on the output result of the feature fusion network.

[0134] In one implementation, when the first sub-feature graph and the second sub-feature graph processed by the first feature fusion network are the same as the third sub-feature graph and the fourth sub-feature graph processed by the second feature fusion network, the processor 1001 inputs the same sub-feature graph into different sub-networks for processing.

[0135] In one implementation, the processor 1001 calls a sub-network of the fourth feature fusion network to process the intermediate fusion result to obtain an activation result corresponding to the intermediate fusion result; calls another sub-network of the fourth feature fusion network to process the regional feature map to obtain an activation result corresponding to the regional feature map; determines a target fusion result based on the intermediate fusion result and the activation results corresponding to the regional feature map, and determines the fusion features of the M sub-feature maps based on the target fusion result.

[0136] In one implementation, the processor 1001 calls the Kth convolution processing subnetwork of the feature extraction network to perform convolution processing on the image to be processed or the intermediate feature map of the image to be processed to obtain a processed intermediate feature map; K is any one of the set values, and the set value is part or all of the positive integers from 1 to X-1; determines the first part of the features and the second part of the features from the processed intermediate feature map; outputs the first part of the features to the K+1th convolution processing subnetwork of the feature extraction network for convolution processing, and outputs the second part of the features to the feature splicing subnetwork; calls the feature splicing subnetwork to determine the initial feature map of the image to be processed based on the second part of the features of the Kth convolution processing subnetwork and the output result of the Xth convolution processing subnetwork.

[0137] In one implementation, the processor 1001 determines a reference area corresponding to the object to be identified in the initial feature map corresponding to each feature extraction network; determines a target area based on the determined multiple reference areas, and obtains a regional feature map corresponding to the object to be identified based on the target area and the initial feature map corresponding to each feature extraction network.

[0138] In one implementation, the processor 1001 determines the placement of the objects to be identified in the image to be processed based on the identified object categories and placement positions of the objects to be identified.

[0139] In a specific implementation, the above-mentioned computer device can execute the above-mentioned functions through its built-in functional modules. Figures 1a to 7 For the implementation methods provided in each step, please refer to the implementation methods provided in the above steps for details, which will not be repeated here.

[0140] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figure 2 For the image processing methods provided in each step, please refer to the implementation methods provided in the above steps, which will not be repeated here.

[0141] The above-mentioned computer-readable storage medium can be the internal storage unit of the recommendation model training device or the above-mentioned terminal device provided in any of the aforementioned embodiments, such as a hard disk or memory of an electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the electronic device. Further, the computer-readable storage medium can also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0142] The terms "first", "second", "third", "fourth", etc. in the claims, specification and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0143] In the specific implementation of the present application, data related to the operation object information (such as the image to be processed, etc.) is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0144] Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. Displaying the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the items listed in association and all possible combinations, and includes these combinations. It can be appreciated by those of ordinary skill in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0145] The method and related device provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application, and can be specifically implemented by computer program instructions for each process and / or box of the method flow chart and / or structural diagram, and the combination of the process and / or box in the flow chart and / or block diagram. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes of the flow chart and / or one box or multiple boxes of the structural diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function specified in one process or multiple processes of the flow chart and / or one box or multiple boxes of the structural diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the structure diagram.

Claims

1. An image processing method, characterized in that: The method comprises: Calling a feature extraction network to perform feature extraction on an image to be processed or an intermediate feature map of the image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes an object to be identified, and the feature extraction network includes a feature splicing subnetwork and X convolution processing subnetworks connected in series, where X is a positive integer greater than 1; Acquire a regional feature map corresponding to the object to be identified based on the initial feature map; The regional feature map is split into M sub-feature maps, and a feature fusion model is called to perform feature fusion processing on the M sub-feature maps to obtain fusion features; M is a positive integer greater than 1; Determining a recognition result of the object to be recognized based on the fusion feature; Wherein, the feature fusion model includes a feature fusion network, and the feature fusion network includes a first subnetwork and a second subnetwork connected in parallel, the first subnetwork corresponds to a first weight and a first activation function, and the second subnetwork corresponds to a second weight and a second activation function; the first subnetwork is used to process input data based on the first weight and the first activation function to obtain an activation result, and the second subnetwork is used to process input data based on the second weight and the second activation function to obtain an activation result, and the first weight and the second weight are two different weights; the output result of the feature fusion network is a fusion result determined based on the activation results output by the first subnetwork and the second subnetwork, and the fusion feature is determined based on the output result of the feature fusion network; The calling of the feature extraction network to perform feature extraction on the image to be processed or the intermediate feature map of the image to be processed to obtain the initial feature map of the image to be processed includes: Calling the Kth convolution processing subnetwork of the feature extraction network to perform convolution processing on the image to be processed or the intermediate feature map of the image to be processed to obtain a processed intermediate feature map; K is any one of the set values, and the set value is part or all of the positive integers from 1 to X-1; Splitting the processed intermediate feature map to obtain a first part of features and a second part of features; Outputting the first part of features to the K+1th convolution processing subnetwork of the feature extraction network for convolution processing, and outputting the second part of features to the feature splicing subnetwork; The feature splicing subnetwork is called to determine an initial feature map of the image to be processed based on the second part of the features of the Kth convolution processing subnetwork and the output result of the Xth convolution processing subnetwork.

2. The method according to claim 1, characterized in that The feature fusion model includes a plurality of feature fusion networks, the plurality of feature fusion networks include a first feature fusion network, a second feature fusion network and a third feature fusion network, the first feature fusion network and the second feature fusion network are connected in parallel, and the input end of the third feature fusion network is connected to the output ends of the first feature fusion network and the second feature fusion network; wherein, calling the feature fusion model to perform feature fusion processing on the M sub-feature graphs to obtain fused features includes: The first sub-network and the second sub-network of the first feature fusion network are called to process the first sub-feature map and the second sub-feature map respectively, and a first fusion result is determined based on the activation results corresponding to the first sub-feature map and the second sub-feature map obtained by the processing; the first sub-feature map and the second sub-feature map are any two different sub-feature maps among the M sub-feature maps; The first sub-network and the second sub-network of the second feature fusion network are called to process the third sub-feature map and the fourth sub-feature map respectively, and a second fusion result is determined based on the activation results corresponding to the third sub-feature map and the fourth sub-feature map obtained by the processing; the third sub-feature map and the fourth sub-feature map are any two different sub-feature maps among the M sub-feature maps; Inputting the first fusion result and the second fusion result into the third feature fusion network for processing to obtain an intermediate fusion result; The fusion features of the M sub-feature maps are determined based on the intermediate fusion result.

3. The method according to claim 2, characterized in that When the first sub-feature graph and the second sub-feature graph processed by the first feature fusion network are the same as the third sub-feature graph and the fourth sub-feature graph processed by the second feature fusion network, the first feature fusion network and the second feature fusion network input the same sub-feature graph into different sub-networks for processing.

4. The method according to claim 2, characterized in that: The multiple feature fusion networks further include a fourth feature fusion network, wherein an input end of the fourth feature fusion network is connected to an output end of the third feature fusion network; and determining the fusion features of the M sub-feature graphs based on the intermediate fusion result includes: Calling a sub-network of the fourth feature fusion network to process the intermediate fusion result to obtain an activation result corresponding to the intermediate fusion result; Calling another sub-network of the fourth feature fusion network to process the regional feature map to obtain an activation result corresponding to the regional feature map; Based on the intermediate fusion result and the activation results respectively corresponding to the regional feature maps, a target fusion result is determined, and based on the target fusion result, fusion features of the M sub-feature maps are determined.

5. The method according to any one of claims 1 to 4, characterized in that The feature extraction network is included in the feature extraction model, and the feature extraction model includes a plurality of the feature extraction networks connected in series, the input data of the Yth feature extraction network is the output result of the Y-1th feature extraction network, Y is a positive integer greater than 1, and each feature extraction network obtains an initial feature map of the image to be processed after processing the input data, and the feature scales of the initial feature maps corresponding to the feature extraction networks are different; Wherein, the acquiring of the regional feature map corresponding to the object to be identified based on the initial feature map includes: Determine a reference area corresponding to the object to be identified in the initial feature map corresponding to each of the feature extraction networks; A target area is determined based on the determined multiple reference areas, and a regional feature map corresponding to the object to be identified is obtained based on the target area and the initial feature maps corresponding to each of the feature extraction networks.

6. The method according to claim 1, characterized in that There are one or more objects to be identified in the image to be processed, and the identification result of each object to be identified includes an object category and a placement position; the method further includes: Based on the identified object categories and placement positions of the objects to be identified, the placement conditions of the objects to be identified in the image to be processed are determined.

7. An image processing device, characterized in that: The device comprises: An extraction module is used to call a feature extraction network to perform feature extraction on an image to be processed or an intermediate feature map of the image to be processed to obtain an initial feature map of the image to be processed, wherein the image to be processed includes an object to be identified, and the feature extraction network includes a feature splicing subnetwork and X convolution processing subnetworks connected in series, where X is a positive integer greater than 1; An acquisition module, configured to acquire a regional feature map corresponding to the object to be identified based on the initial feature map; A fusion module is used to split the regional feature map into M sub-feature maps, and call a feature fusion model to perform feature fusion processing on the M sub-feature maps to obtain fusion features, where M is a positive integer greater than 1; A determination module, used to determine the recognition result of the object to be recognized based on the fusion feature; The determination module is further used to call the Kth convolution processing subnetwork of the feature extraction network to perform convolution processing on the image to be processed or the intermediate feature map of the image to be processed to obtain a processed intermediate feature map; K is any one of the set values, and the set value is part or all of the positive integers from 1 to X-1; split the first part of features and the second part of features from the processed intermediate feature map; The extraction module is further used to output the first part of features to the K+1th convolution processing subnetwork of the feature extraction network for convolution processing, and output the second part of features to the feature splicing subnetwork; The determination module is further used to call the feature splicing subnetwork to determine the initial feature map of the image to be processed based on the second part of the features of the Kth convolution processing subnetwork and the output result of the Xth convolution processing subnetwork; Among them, the feature fusion model includes a feature fusion network, and the feature fusion network includes a first subnetwork and a second subnetwork connected in parallel, the first subnetwork corresponds to a first weight and a first activation function, and the second subnetwork corresponds to a second weight and a second activation function; the first subnetwork is used to process the input data based on the first weight and the first activation function to obtain an activation result, and the second subnetwork is used to process the input data based on the second weight and the second activation function to obtain an activation result, and the first weight and the second weight are two different weights; the output result of the feature fusion network is a fusion result determined based on the activation results output by the first subnetwork and the second subnetwork, and the fusion feature is determined based on the output result of the feature fusion network.

8. A computer device, characterized in that: include: A processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are connected to each other, wherein the memory stores an executable program code, and the processor is used to call the executable program code to implement the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to implement the image processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images