Image recognition method

Abstract

The invention discloses an image recognition method which is applied to a terminal, and the specific technical scheme comprises the steps: obtaining a to-be-recognized image, obtaining the terminal reasoning time, the server reasoning time and the information transmission time, and carrying out the recognition of the to-be-recognized image based on the terminal reasoning time, the server reasoning time and the information transmission time. Determining a performance gap between the server and the terminal for processing the sample image; under the condition that the performance difference is greater than or equal to a preset judgment threshold value, calculating a first quantity characteristic graph processed by the terminal and a second quantity characteristic graph processed by the server according to the terminal reasoning time, the server reasoning time and the information transmission time, and identifying the first quantity characteristic graph to obtain a first identification result; obtaining a second identification result obtained by identifying the second quantity feature map by the server; and determining the category of the to-be-recognized image based on the first recognition result and the second recognition result. Therefore, the image recognition efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of information security technology, and in particular, relates to an image recognition method. Background Art

[0002] When using a terminal to recognize an image, due to limited resources in the terminal, when performing image recognition on a complex image, the terminal's efficiency in recognizing the image is low due to insufficient computing power and limited storage resources. Summary of the invention

[0003] The embodiments of the present application provide a method, device, equipment, storage medium and program product for image recognition generation, which can reduce the amount of calculation of a terminal and a server, thereby improving the efficiency of image recognition.

[0004] In a first aspect, an embodiment of the present application provides an image recognition method, applied to a terminal, the method comprising:

[0005] Obtain an image to be recognized, a terminal reasoning time, a server reasoning time, and an information transmission time, wherein the terminal reasoning time is the time it takes for the terminal to recognize a sample image, the server reasoning time is the time it takes for the server to recognize the sample image, and the information transmission time is the time it takes for the terminal to send the sample image to the server;

[0006] Determine, based on the terminal reasoning time, the server reasoning time, and the information transmission time, a performance gap between the server and the terminal in processing the sample image;

[0007] When the performance gap is greater than or equal to the preset judgment threshold, a first quantity feature map processed by the terminal and a second quantity feature map processed by the server are calculated according to the terminal reasoning time, the server reasoning time and the information transmission time, and the total number of the first quantity feature map and the second quantity feature map is the same as the number of feature maps included in the sample image;

[0008] Identify the first quantitative feature graph to obtain a first identification result;

[0009] Sending the second quantitative feature map to the server;

[0010] Receive a second recognition result fed back by the server, where the second recognition result is obtained by the server through recognition of the second quantitative feature graph;

[0011] The category of the image to be recognized is determined based on the first recognition result and the second recognition result.

[0012] In a possible implementation, determining the performance gap between the server and the terminal in processing the sample image based on the terminal reasoning time, the server reasoning time, and the information transmission time includes:

[0013] The performance gap is calculated according to the following formula:

[0014] δ=t1-(t2+t3)

[0015] Among them, δ represents the performance gap, t1 is the terminal reasoning time, t2 is the information transmission time, and t3 is the server reasoning time.

[0016] In a possible implementation, the method further includes:

[0017] When the performance gap is smaller than the preset judgment threshold, the image to be recognized is input into the image recognition model to obtain the category of the image to be recognized.

[0018] In a possible implementation manner, the calculating, according to the terminal reasoning time, the server reasoning time, and the information transmission time, a first quantity characteristic graph of terminal processing and a second quantity characteristic graph of server processing includes:

[0019] According to the preset algorithm, based on the terminal reasoning time, the server reasoning time and the information transmission time, the first quantity and the second quantity are calculated.

[0020] In a possible implementation manner, the calculating the first quantity and the second quantity according to the preset algorithm based on the terminal reasoning time, the server reasoning time, and the information transmission time includes:

[0021] The first quantity is calculated according to the following formula:

[0022]

[0023] Among them, N mobile is the first quantity, T server is the server reasoning time, T tran is the information transmission time, T mobile is the terminal reasoning time, N total The total number of feature maps obtained by the terminal from performing feature extraction on the image to be identified;

[0024] N server =N total -N mobile

[0025] Among them, N serveris the second number.

[0026] In a possible implementation manner, after identifying the first quantitative feature graph and obtaining a first identification result, the method further includes:

[0027] Determining whether the highest confidence level in the first recognition result is less than a first preset confidence threshold;

[0028] Sending the second quantity feature graph to the server includes:

[0029] When the highest confidence in the first recognition result is less than the first preset confidence threshold, the second quantitative feature map is sent to the server.

[0030] In a possible implementation, the method further includes:

[0031] When the highest confidence in the first recognition result is greater than or equal to a first preset confidence threshold, the category corresponding to the highest confidence in the first recognition result is used as the category of the image to be recognized.

[0032] In a possible implementation manner, determining the category of the image to be recognized based on the first recognition result and the second recognition result includes:

[0033] Determining whether the highest confidence level in the second recognition result is less than a second preset confidence threshold;

[0034] When the highest confidence in the second recognition result is less than the second preset confidence threshold, summing the confidences of the same category in the first recognition result and the second recognition result to obtain a combined confidence for each category;

[0035] The category with the highest combined confidence is used as the category of the image to be identified.

[0036] In a possible implementation, the method further includes:

[0037] When the highest confidence in the second recognition result is greater than or equal to the second preset confidence threshold, the category corresponding to the highest confidence in the second recognition result is used as the category of the image to be recognized.

[0038] In a possible implementation manner, before acquiring the image to be recognized, the method further includes:

[0039] Get a sample image;

[0040] Inputting the sample image into an image recognition model, and recording the terminal inference time when the image recognition model recognizes the sample image;

[0041] sending the sample image to the server;

[0042] A recognition response fed back by the server is received, where the recognition response includes a server reasoning time for the server to recognize the sample image.

[0043] In a second aspect, an embodiment of the present application provides an image recognition device, which is applied to a terminal, and the device includes:

[0044] An acquisition module, used to acquire an image to be recognized, a terminal reasoning time, a server reasoning time, and an information transmission time. The terminal reasoning time is the time it takes for the terminal to recognize a sample image, the server reasoning time is the time it takes for the server to recognize the sample image, and the information transmission time is the time it takes for the terminal to send the sample image to the server;

[0045] A determination module, configured to determine a performance gap between the server and the terminal in processing a sample image based on the terminal reasoning time, the server reasoning time, and the information transmission time;

[0046] a calculation module, configured to calculate, when the performance gap is greater than or equal to the preset judgment threshold, a first quantity feature map processed by the terminal and a second quantity feature map processed by the server according to the terminal reasoning time, the server reasoning time, and the information transmission time, wherein the total number of the first quantity feature map and the second quantity feature map is the same as the number of feature maps included in the sample image;

[0047] An identification module, used to identify the first quantitative feature map and obtain a first identification result;

[0048] A sending module, used for sending the second quantitative feature map to a server;

[0049] A receiving module, configured to receive a second recognition result fed back by the server, where the second recognition result is obtained by the server through recognition of the second quantitative feature graph;

[0050] A determination module is used to determine the category of the image to be recognized based on the first recognition result and the second recognition result.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0052] When the processor executes the computer program instructions, the image recognition method as described in any one of the first aspects is implemented.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for image recognition as described in any one of the first aspects is implemented.

[0054] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device performs the image recognition method as described in any one of the first aspects.

[0055] In an image recognition method, apparatus, device, storage medium and program product of an embodiment of the present application, a terminal obtains the terminal reasoning time, the server reasoning time and the information transmission time. Since the terminal reasoning time, the server reasoning time and the information transmission time can reflect the performance of the terminal and the server in processing sample images respectively, the terminal can determine the performance gap between the terminal and the server according to the terminal reasoning time, the server reasoning time and the information transmission time. When the performance gap is greater than or equal to the preset judgment threshold, it indicates that the performance of the terminal in processing the sample image is poor. In order to improve the processing efficiency of the image to be identified, the terminal and the server can be used in combination to process the image to be identified. In this way, the terminal allocates a first quantity feature map to the terminal and a second quantity feature map to the server according to the terminal reasoning time, the server reasoning time and the information transmission time. The terminal receives the second recognition result fed back by the server, and the category corresponding to the image to be identified can be determined by combining the recognition result of the terminal on the first quantity feature map and the second recognition result fed back by the server, thereby improving the accuracy of the recognition result. In this way, the terminal and the server only need to identify part of the feature map, avoiding the processing of all feature maps corresponding to the image to be identified, reducing the amount of calculation of the terminal and the server, thereby improving the efficiency of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 It is a flowchart of an image recognition method provided by an embodiment of the present application;

[0058] Figure 2 It is a flowchart of a method for obtaining time information provided in an embodiment of the present application;

[0059] Figure 3 is an exemplary schematic diagram of an image recognition method provided in an embodiment of the present application;

[0060] Figure 4 is an exemplary schematic diagram of another image recognition method provided in an embodiment of the present application;

[0061] Figure 5 This is an exemplary schematic diagram of another image recognition method provided in an embodiment of the present application.

[0062] Figure 6 is a structural schematic diagram of an image recognition device provided in an embodiment of the present application;

[0063] Figure 7 It is a structural schematic diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0064] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0065] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0066] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, device, storage medium and program product for image recognition. Figure 1 An image recognition method provided in an embodiment of the present application is introduced, and the method is applied to a terminal, wherein the terminal may be a smart phone, a tablet computer, or an Internet of Things device. Figure 1 As shown, the method includes:

[0067] S101, obtaining an image to be recognized, terminal reasoning time, server reasoning time, and information transmission time.

[0068] Among them, the terminal inference time is the time it takes for the terminal to recognize the sample image, the server inference time is the time it takes for the server to recognize the sample image, the information transmission time is the time it takes for the terminal to send the sample image to the server, and the image to be recognized includes multiple feature maps.

[0069] In the embodiment of the present application, the sample image and the image to be identified are from the same batch of images, wherein the sample image is any image in a batch of images to be identified.

[0070] It should be noted that each image in the same batch of images contains a similar amount of information and the scenes corresponding to the image contents are similar. For example, the same batch of images may be images of different animals or vehicles.

[0071] Specifically, the terminal may receive multiple images sent by a third-party device or a third-party platform, wherein the multiple images are a batch of images.

[0072] After the terminal obtains the image to be recognized, the image is gray-scaled, and then a feature extraction algorithm is used to extract a feature map of the gray-scale image to obtain multiple feature maps included in the image to be recognized. Among them, the terminal can load a lightweight convolutional neural network MobileNet and use the convolutional neural network to extract the feature map of the image to be recognized.

[0073] S102: Determine the performance gap between the server and the terminal in processing sample images based on the terminal reasoning time, the server reasoning time, and the information transmission time.

[0074] It can be understood that the shorter the time it takes for the terminal to process the sample image, the better the performance of the terminal in processing the sample image, and the shorter the time it takes for the server to process the sample image, the better the performance of the server in processing the sample image. Therefore, the terminal can determine the performance of the terminal in processing the sample image and the performance of the server in processing the sample image based on the terminal reasoning time, the server reasoning time, and the information transmission time. Further, the terminal can determine the performance gap between the server and the terminal in processing the sample image based on the performance of the terminal in processing the sample image and the performance of the server in processing the sample image.

[0075] S103. When the performance gap is greater than or equal to a preset judgment threshold, a first quantity characteristic graph of terminal processing and a second quantity characteristic graph of server processing are calculated according to the terminal reasoning time, the server reasoning time and the information transmission time.

[0076] The total number of the first quantity feature map and the second quantity feature map is the same as the number of feature maps included in the sample image, and the preset judgment threshold is pre-set based on experience.

[0077] It should be noted that when the performance gap is greater than or equal to the preset judgment threshold, it means that the performance of the terminal in processing the sample image is much lower than the performance of the server in processing the sample image. Therefore, in order to improve the image processing efficiency, the terminal and the server can be used in combination to perform image recognition on the image to be identified at the same time.

[0078] S104: Identify the first quantitative feature graph to obtain a first identification result.

[0079] Specifically, an image recognition model is deployed in the terminal, and a first number of feature maps are recognized using the image recognition model to obtain a first recognition result.

[0080] S105. Send a second quantitative feature map to the server.

[0081] S106: Receive the second recognition result fed back by the server.

[0082] Among them, the second recognition result is obtained by the server identifying the second quantitative feature graph.

[0083] In one example, a deep residual network (Residual Network, ResNet) is deployed in the server, and the second quantitative feature map can be recognized by ResNet to obtain a second recognition result.

[0084] S107: Determine the category of the image to be recognized based on the first recognition result and the second recognition result.

[0085] By adopting the above method, the terminal obtains the terminal reasoning time, the server reasoning time and the information transmission time. Since the terminal reasoning time, the server reasoning time and the information transmission time can reflect the performance of the terminal and the server in processing the sample image respectively, the terminal can determine the performance gap between the terminal and the server according to the terminal reasoning time, the server reasoning time and the information transmission time. When the performance gap is greater than or equal to the preset judgment threshold, it means that the performance of the terminal in processing the sample image is poor. In order to improve the processing efficiency of the image to be identified, the terminal and the server can be used in combination to process the image to be identified. In this way, the terminal allocates the first quantity feature map to the terminal and the second quantity feature map to the server according to the terminal reasoning time, the server reasoning time and the information transmission time. The terminal receives the second recognition result fed back by the server, and the category corresponding to the image to be identified can be determined by combining the recognition result of the terminal on the first quantity feature map and the second recognition result fed back by the server, thereby improving the accuracy of the recognition result. In this way, the terminal and the server only need to identify part of the feature map, avoiding the processing of all feature maps corresponding to the image to be identified, reducing the amount of calculation of the terminal and the server, thereby improving the image recognition efficiency.

[0086] Before obtaining the image to be identified, the terminal reasoning time, the server reasoning time and the information transmission time in S101, the terminal calculates the terminal reasoning time, the server reasoning time and the information transmission time corresponding to the sample image. Figure 2 As shown, the method includes:

[0087] S201: Obtain a sample image.

[0088] The sample image and the image to be identified are from the same batch of images, and the sample image is any image in a batch of images to be identified.

[0089] It should be noted that each image in the same batch of images contains a similar amount of information and the scenes corresponding to the image contents are similar. For example, the same batch of images may be images of different animals.

[0090] S202: Input the sample image into the image recognition model, and record the terminal inference time of the image recognition model recognizing the sample image.

[0091] S203: Send the sample image to the server.

[0092] The terminal records the timestamp of sending the sample image to the server, and records the timestamp of receiving the response fed back by the server, and calculates the difference between the timestamp of receiving the response and the timestamp of sending the sample image to obtain the information transmission time. The receiving response is sent by the server to the terminal after receiving the sample image sent by the terminal.

[0093] S204: Receive an identification response fed back by the server.

[0094] The recognition response includes the server inference time for the server to recognize the sample image.

[0095] By using the method provided in the embodiment of the present application, before the terminal recognizes the image to be recognized, the terminal can obtain the terminal reasoning time corresponding to the sample image and the server reasoning time fed back by the server using the sample image. In this way, after the performance gap is determined using the terminal reasoning time, information transmission time and server reasoning time corresponding to the sample image, the processing method for the image to be recognized that belongs to the same batch of images as the sample image can be determined based on the performance gap, thereby ensuring the recognition efficiency of the image to be recognized.

[0096] Specifically, for the above S102, based on the terminal reasoning time, the server reasoning time and the information transmission time, the performance difference between the server and the terminal in processing the sample image is determined. There are three implementation methods:

[0097] In the first implementation, the terminal can use Formula 1 to calculate the performance gap:

[0098] δ=t1-(t2+t3) Formula 1

[0099] Among them, δ represents the performance gap, t1 is the terminal reasoning time, t2 is the information transmission time, and t3 is the server reasoning time.

[0100] Specifically, when δ<α and α≤0, the image to be identified is input into the image recognition model in the terminal to obtain the category of the image to be identified, where α is a non-positive hyperparameter, that is, the preset judgment threshold.

[0101] In a second implementation, the terminal may determine the performance gap based on a terminal inference time threshold.

[0102] Specifically, the terminal determines whether the terminal reasoning time is greater than or equal to the terminal reasoning time threshold. When the terminal reasoning time is greater than or equal to the terminal reasoning time threshold, it means that the terminal reasoning time is large enough and the performance gap between the terminal and the server in processing sample images is large enough. Therefore, the server and the terminal are used in combination to process the image to be recognized.

[0103] In a third implementation, the terminal may determine the performance gap based on a terminal reasoning time threshold and a server reasoning time threshold.

[0104] Specifically, the terminal determines whether the terminal reasoning time is greater than or equal to the terminal reasoning time threshold, and whether the server reasoning time is less than the server reasoning time threshold. When the terminal reasoning time is greater than or equal to the terminal reasoning time threshold, and the server reasoning time is less than the server reasoning time threshold, it means that the terminal reasoning time is large enough, and the server reasoning time is small enough, thereby ensuring that the performance gap between the terminal and the server in processing the sample image is large enough, and therefore, the server and the terminal are used in combination to process the image to be recognized.

[0105] In some embodiments of the present application, the first quantity characteristic graph of terminal processing and the second quantity characteristic graph of server processing are calculated according to the obtained terminal reasoning time, server reasoning time and information transmission time in S103 above, which can be implemented as follows:

[0106] According to a preset algorithm, based on the terminal reasoning time, the server reasoning time and the information transmission time, the first quantity and the second quantity are calculated.

[0107] The first quantity can be calculated according to Formula 2:

[0108]

[0109] Among them, N mobile is the first quantity, T server is the server reasoning time, T tranis the information transmission time, T mobile is the terminal reasoning time, N total The total number of feature maps obtained by the terminal when performing feature extraction on the image to be recognized.

[0110] The second quantity is calculated according to formula 3:

[0111] N server =N total -N mobile Formula 3

[0112] Among them, N server is the second quantity.

[0113] In this way, the server reasoning time, the terminal reasoning time and the information transmission time can effectively reflect the image recognition efficiency and image recognition capability of the terminal and the server. In order to ensure the overall efficiency of image recognition, the first quantity and the second quantity calculated according to the server reasoning time, the terminal reasoning time and the information transmission time can effectively utilize the image recognition capability of the terminal and the server, thereby ensuring the overall image recognition efficiency.

[0114] After the first number and the second number are obtained by the above calculation, the terminal can shuffle the order of the identified feature maps, and the terminal randomly selects the first number of feature maps from the identified feature maps to ensure that the terminal can obtain sufficient information from the first number of feature maps for image recognition. Similarly, the terminal sends the second number of feature maps to the server so that the server can obtain sufficient information from the second number of feature maps for image recognition.

[0115] In the process of the above-mentioned terminal identifying the first quantitative feature map, the terminal device can use a deep separable convolutional network to perform a convolution operation on the first quantitative feature map, output a Vector result vector, set a fully connected layer in the convolutional network, and identify the categories included in the image to be identified, as well as the confidence corresponding to each category.

[0116] After the terminal identifies the first quantitative feature graph and obtains the first identification result, the terminal may judge the first identification result to determine whether the highest confidence level in the first identification result is less than a first preset confidence threshold.

[0117] The first preset confidence threshold is preset based on experience.

[0118] Specifically, Formula 4 is used to determine whether the highest confidence level is less than the first preset confidence level threshold:

[0119]

[0120] in, That is, it is the highest confidence, and β is the first preset confidence threshold.

[0121] When using formula 4 for judgment, the judgment results obtained include the following two situations:

[0122] Case 1: When the highest confidence in the first recognition result is less than the first preset confidence threshold, a second quantitative feature map is sent to the server.

[0123] It can be understood that when the highest confidence is less than the first preset confidence threshold, it means that the accuracy of the first recognition result is low, and the terminal can further obtain the second recognition result from the server.

[0124] Case 2: When the highest confidence in the first recognition result is greater than or equal to the first preset confidence threshold, the category corresponding to the highest confidence in the first recognition result is used as the category of the image to be recognized.

[0125] It can be understood that when the highest confidence is greater than or equal to the first preset confidence threshold, it means that the accuracy of the first recognition result is high. Therefore, the terminal can directly use the category corresponding to the highest confidence in the first recognition result as the category of the image to be recognized.

[0126] It can be understood that the above-mentioned terminal identifies the first quantitative feature map to obtain the first recognition result. The terminal determines whether the highest confidence in the first recognition result is less than the first preset confidence threshold. If the highest confidence in the first recognition result is less than the first preset confidence threshold, it means that the accuracy of the first recognition result is insufficient. The terminal can further refer to the second recognition result of the server to determine the image category. In this way, the accuracy of the recognition result can be improved. Similarly, if the highest confidence in the first recognition result is greater than or equal to the first preset confidence threshold, it means that the accuracy of the first recognition result is high. The category corresponding to the highest confidence in the first recognition result can be used as the category of the image to be recognized, thereby ensuring the accuracy of the image recognition result.

[0127] After the terminal receives the second recognition result sent by the server, it can also determine whether the category with the highest confidence in the second recognition result can be used as the category of the image to be recognized based on the highest confidence in the second recognition result. Based on this, the above S107, determining the category of the image to be recognized based on the first recognition result and the second recognition result, can be implemented as follows:

[0128] It is determined whether the highest confidence level in the second recognition result is less than a second preset confidence threshold.

[0129] Specifically, Formula 5 is used to determine whether the highest confidence in the second recognition result is less than the first preset confidence threshold:

[0130]

[0131] in, That is, β+γ is the second preset confidence threshold. γ>0, γ is the preset hyperparameter.

[0132] The judgment results obtained according to the above formula 5 specifically include the following two situations:

[0133] Case 1: When the highest confidence in the second recognition result is less than the second preset confidence threshold, the confidences of the same category in the first recognition result and the second recognition result are summed to obtain a combined confidence for each category, and the category with the highest combined confidence is used as the category of the image to be recognized.

[0134] It can be understood that when the highest confidence in the above-mentioned second recognition result is less than the second preset confidence threshold, it means that the accuracy of the second recognition result is low. Therefore, the terminal can determine the category of the image to be recognized by combining the first recognition result and the second recognition result.

[0135] Specifically, the calculation of the above combined confidence is as shown in Formula 6:

[0136]

[0137] Among them, K is the combined confidence, are the confidences of the same category in the first recognition result and the second recognition result respectively.

[0138] Then, based on calculating the combined confidence of each category, the terminal determines the category with the highest combined confidence among the combined confidences by using Formula 7:

[0139] Out=max (K) Formula 6

[0140] Among them, Out means merging the category with the highest confidence.

[0141] Case 2: when the highest confidence in the second recognition result is greater than or equal to the second preset confidence threshold, the category corresponding to the highest confidence in the second recognition result is used as the category of the image to be recognized.

[0142] It can be understood that when the highest confidence in the second recognition result is greater than or equal to the second preset confidence threshold, it means that the accuracy of the second recognition result is higher. Therefore, the terminal can directly use the category corresponding to the highest confidence in the second recognition result as the category of the image to be recognized.

[0143] It can be understood that when the above-mentioned terminal receives the second recognition result sent by the server, the terminal determines whether the highest confidence in the second recognition result is less than the second preset confidence threshold. If the highest confidence in the second recognition result is less than the second preset confidence threshold, it means that the accuracy of the second recognition result is insufficient. The terminal combines the first recognition result and the second recognition result to determine the image category. In this way, the accuracy of the recognition result can be improved. Similarly, if the highest confidence in the second recognition result is greater than or equal to the second preset confidence threshold, it means that the second recognition result has a high accuracy. The category corresponding to the highest confidence in the second recognition result can be used as the category of the image to be recognized, thereby ensuring the accuracy of the image recognition result.

[0144] The following combination Figure 3 The process of the image recognition method provided in the embodiment of the present application is introduced as follows: Figure 2 As shown, the method specifically includes:

[0145] S301, time consumption calculation.

[0146] Among them, the time consumption calculation refers to the process in which the terminal records the above-mentioned terminal reasoning time, server reasoning time and information transmission time, and then calculates the performance gap based on the terminal reasoning time, server reasoning time and information transmission time.

[0147] S302. Select a plan.

[0148] Among them, the selection scheme refers to the terminal selecting an image recognition scheme according to the performance gap and the preset judgment threshold. The specific content refers to the relevant description in the above embodiment and will not be repeated here.

[0149] S303: Determine to use a dynamic distributed network architecture for image recognition.

[0150] Among them, the use of a dynamic distributed network architecture for image recognition is a process in which the terminal calculates a first quantity feature map corresponding to the terminal and a second quantity feature map corresponding to the server based on the terminal inference time, the server inference time and the information transmission time, and determines the image category based on a first recognition result of the terminal for the first quantity feature map and a second recognition result of the server for the second quantity feature map.

[0151] S304: Deploy algorithms and applications.

[0152] Specifically, the terminal locally deploys a corresponding image recognition model, and a method for determining an image category based on a first recognition result and a second recognition result. The server locally deploys a corresponding image recognition model.

[0153] For distributed network architecture, the specific calculation method is as follows: Figure 4 As shown, the method includes:

[0154] S3031. Perform image pre-processing.

[0155] The image pre-processing refers to the terminal device performing grayscale processing on the image to be identified, and performing feature extraction on the grayscale image after the grayscale processing to obtain a plurality of feature maps included in the image to be identified.

[0156] S3032, disrupt the channel.

[0157] Specifically, scrambling the channels means that the terminal scrambles the arrangement order of the multiple feature maps obtained above, wherein each feature map corresponds to a channel.

[0158] S3033, split channel.

[0159] Specifically, the feature graph processed by the terminal and the feature graph processed by the server are calculated according to the above method for calculating the first quantity and the second quantity.

[0160] S3034. The terminal uses an algorithm model to identify the image to be identified.

[0161] The algorithm model is the image recognition model in the above embodiment. Specifically, the embodiment of the present application does not limit the category of the image recognition model.

[0162] S3035. The server uses an algorithm model to identify the image to be identified.

[0163] S3036. Output the final result.

[0164] Specifically, the method for the terminal to determine the image category is referred to the relevant description in the above embodiment, which will not be repeated here.

[0165] By adopting the method provided in the embodiment of the present application, on the basis of calculating the time consumption, the optimal image recognition scheme is determined according to the time consumption, and a corresponding number of feature maps are allocated to the server and the terminal. In this way, the image recognition performance of the terminal and the server can be determined by the time consumption, and then the corresponding number of feature maps are allocated according to the image recognition performance, so that the optimal image recognition scheme can be achieved, thereby improving the image recognition efficiency and enhancing the user experience. In addition, since a lightweight model is deployed on the terminal, the computing resources and storage space of the terminal can be effectively utilized while performing image recognition, further improving the image recognition efficiency, reducing energy consumption, and extending the battery life. In addition, since the server only needs to process the second number of feature maps, the server cannot obtain all the image information, thereby effectively protecting the information security of the terminal.

[0166] exist Figure 4 Based on the embodiment shown, combined Figure 5 Introduce a method for recognizing images using the above distributed network architecture, such as Figure 5 As shown, after acquiring the image to be identified, the terminal processes the image to be identified into a grayscale image, and then extracts the feature map of the grayscale image to obtain N feature maps. Then, the terminal scrambles the order of the multiple feature maps, and uses the above method for calculating the first number and the second number to calculate the number of feature maps processed by the terminal and the number of feature maps processed by the server.

[0167] The terminal inputs the feature map of n channels into the image recognition model, namely the depthwise separable convolution network, to obtain a vector. A fully connected layer is connected after the convolutional neural network, and the vector is input into the fully connected layer, which outputs the first recognition result.

[0168] The terminal determines whether the first recognition result meets the confidence requirement, that is, whether the highest confidence in the first recognition result is greater than a first preset confidence threshold.

[0169] When the highest confidence level in the first recognition result meets the confidence level requirement, the terminal outputs the category corresponding to the highest confidence level in the first recognition result.

[0170] When the highest confidence in the first recognition result does not meet the confidence requirement, the terminal sends the feature map of Nn channels to the server, and the server inputs the feature map of Nn channels into the deep residual network (High ResNetConvolution) to obtain a second recognition result, and the server feeds the second recognition result back to the terminal.

[0171] The terminal determines the final output result based on the first recognition result and the second recognition result, and determines the category of the image to be recognized. Specifically, the method for the terminal to determine the category of the image to be recognized based on the first recognition result and the second recognition result refers to the relevant description in the above embodiment, which will not be repeated here.

[0172] Based on the same concept, the embodiment of the present application also provides an image recognition device, which is applied to a terminal, such as Figure 6 As shown, the device comprises:

[0173] An acquisition module 601 is used to acquire an image to be recognized, a terminal reasoning time, a server reasoning time, and an information transmission time. The terminal reasoning time is the time it takes for the terminal to recognize a sample image, the server reasoning time is the time it takes for the server to recognize the sample image, and the information transmission time is the time it takes for the terminal to send the sample image to the server.

[0174] A determination module 602, configured to determine a performance difference between the server and the terminal in processing a sample image based on the terminal reasoning time, the server reasoning time, and the information transmission time;

[0175] A calculation module 603 is used to calculate, when the performance gap is greater than or equal to a preset judgment threshold, a first quantity feature map processed by the terminal and a second quantity feature map processed by the server according to the terminal reasoning time, the server reasoning time and the information transmission time, wherein the total number of the first quantity feature map and the second quantity feature map is the same as the number of feature maps included in the sample image;

[0176] An identification module 604 is used to identify the first quantitative feature graph and obtain a first identification result;

[0177] A sending module 605 is used to send the second quantitative feature map to a server;

[0178] A receiving module 606 is configured to receive a second recognition result fed back by the server, where the second recognition result is obtained by the server through recognition of the second quantitative feature graph;

[0179] The determination module 602 is used to determine the category of the image to be recognized based on the first recognition result and the second recognition result.

[0180] In a possible implementation, the determination module 602 is specifically configured to:

[0181] The performance gap is calculated according to the following formula:

[0182] δ=t1-(t2+t3)

[0183] Among them, δ represents the performance gap, t1 is the terminal reasoning time, t2 is the information transmission time, and t3 is the server reasoning time.

[0184] In a possible implementation, the determining module 602 is further configured to:

[0185] When the performance gap is smaller than the preset judgment threshold, the image to be recognized is input into the image recognition model to obtain the category of the image to be recognized.

[0186] In a possible implementation, the calculation module 603 is specifically configured to:

[0187] According to the preset algorithm, based on the terminal reasoning time, the server reasoning time and the information transmission time, the first quantity and the second quantity are calculated.

[0188] In a possible implementation, the calculation module 603 is specifically configured to:

[0189] The first quantity is calculated according to the following formula:

[0190]

[0191] Among them, N mobile is the first quantity, T server is the server reasoning time, T tran is the information transmission time, T mobile is the terminal reasoning time, N total The total number of feature maps obtained by the terminal from performing feature extraction on the image to be identified;

[0192] The second amount is calculated according to the following formula:

[0193] N server =N total -N mobile

[0194] Among them, N server is the second number.

[0195] In a possible implementation, the device further includes:

[0196] A judgment module, used to judge whether the highest confidence in the first recognition result is less than a first preset confidence threshold;

[0197] The sending module 605 is specifically used for:

[0198] When the highest confidence in the first recognition result is less than the first preset confidence threshold, the second quantitative feature map is sent to the server.

[0199] In a possible implementation, the determining module 602 is further configured to:

[0200] When the highest confidence in the first recognition result is greater than or equal to a first preset confidence threshold, the category corresponding to the highest confidence in the first recognition result is used as the category of the image to be recognized.

[0201] In a possible implementation, the determination module 602 is specifically configured to:

[0202] Determining whether the highest confidence level in the second recognition result is less than a second preset confidence threshold;

[0203] When the highest confidence in the second recognition result is less than the second preset confidence threshold, summing the confidences of the same category in the first recognition result and the second recognition result to obtain a combined confidence for each category;

[0204] The category with the highest combined confidence is used as the category of the image to be identified.

[0205] In a possible implementation, the determining module 602 is further configured to:

[0206] When the highest confidence in the second recognition result is greater than or equal to the second preset confidence threshold, the category corresponding to the highest confidence in the second recognition result is used as the category of the image to be recognized.

[0207] In a possible implementation, the device further includes:

[0208] An acquisition module 601 is used to acquire a sample image before acquiring the image to be recognized, the terminal reasoning time, the server reasoning time and the information transmission time;

[0209] An input module, used to input the sample image into an image recognition model, and record the terminal reasoning time when the image recognition model recognizes the sample image;

[0210] The sending module 605 is further used to send the sample image to the server;

[0211] The receiving module 606 is further configured to receive a recognition response fed back by the server, wherein the recognition response includes a server inference time for the server to recognize the sample image.

[0212] It should be noted that the image recognition device is a device corresponding to the above-mentioned image recognition method, and all implementation methods in the above-mentioned method embodiment are applicable to the embodiment of the device and can achieve the same technical effect.

[0213] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0214] The electronic device may include a processor 701 and a memory 702 storing computer program instructions.

[0215] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0216] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 702 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory.

[0217] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0218] The processor 701 implements any one of the image recognition methods in the above embodiments by reading and executing computer program instructions stored in the memory 702 .

[0219] In one example, the electronic device may further include a communication interface 703 and a bus 704. Figure 7 As shown, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 704 and communicate with each other.

[0220] The communication interface 703 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0221] Bus 704 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 704 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0222] In addition, in combination with the image recognition method in the above embodiment, the embodiment of the present application can provide a computer storage medium to implement. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the image recognition methods in the above embodiment is implemented.

[0223] In combination with the image recognition method in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any image recognition method in the above embodiments.

[0224] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0225] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0226] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0227] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0228] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for image recognition, characterized in that: Applied to a terminal, the method comprises: Obtain an image to be recognized, a terminal reasoning time, a server reasoning time, and an information transmission time, wherein the terminal reasoning time is the time it takes for the terminal to recognize a sample image, the server reasoning time is the time it takes for the server to recognize the sample image, and the information transmission time is the time it takes for the terminal to send the sample image to the server; Determine, based on the terminal reasoning time, the server reasoning time, and the information transmission time, a performance gap between the server and the terminal in processing the sample image; When the performance gap is greater than or equal to a preset judgment threshold, a first quantity feature map processed by the terminal and a second quantity feature map processed by the server are calculated according to the terminal reasoning time, the server reasoning time and the information transmission time, and the total number of the first quantity feature map and the second quantity feature map is the same as the number of feature maps included in the sample image; Identify the first quantitative feature graph to obtain a first identification result; Sending the second quantitative feature map to the server; Receive a second recognition result fed back by the server, where the second recognition result is obtained by the server through recognition of the second quantitative feature graph; The category of the image to be recognized is determined based on the first recognition result and the second recognition result.

2. The method according to claim 1, characterized in that The determining, based on the terminal reasoning time, the server reasoning time, and the information transmission time, a performance gap between the server and the terminal in processing the sample image includes: The performance gap is calculated according to the following formula: δ=t1-(t2+t3) Among them, δ represents the performance gap, t1 is the terminal reasoning time, t2 is the information transmission time, and t3 is the server reasoning time.

3. The method according to claim 1, characterized in that Also includes: When the performance gap is smaller than the preset judgment threshold, the image to be recognized is input into an image recognition model to obtain the category of the image to be recognized.

4. The method according to claim 1, characterized in that: The calculating, according to the terminal reasoning time, the server reasoning time and the information transmission time, the first quantity characteristic graph of the terminal processing and the second quantity characteristic graph of the server processing comprises: According to a preset algorithm, the first quantity and the second quantity are calculated based on the terminal reasoning time, the server reasoning time and the information transmission time.

5. The method according to claim 4, characterized in that The calculating the first quantity and the second quantity according to a preset algorithm based on the terminal reasoning time, the server reasoning time, and the information transmission time includes: The first quantity is calculated according to the following formula: Among them, N mobile is the first quantity, T server is the server reasoning time, T tran is the information transmission time, T mobile is the terminal reasoning time, N total The total number of feature maps obtained by the terminal from performing feature extraction on the image to be identified; The second amount is calculated according to the following formula: N server =N total -N mobile Among them, N server is the second number.

6. The method according to claim 1, characterized in that After identifying the first quantitative feature graph and obtaining a first identification result, the method further includes: Determining whether the highest confidence level in the first recognition result is less than a first preset confidence threshold; Sending the second quantity feature graph to the server includes: When the highest confidence in the first recognition result is less than the first preset confidence threshold, the second quantitative feature map is sent to the server.

7. The method according to claim 6, characterized in that Also includes: When the highest confidence in the first recognition result is greater than or equal to a first preset confidence threshold, the category corresponding to the highest confidence in the first recognition result is used as the category of the image to be recognized.

8. The method according to claim 1, characterized in that The determining the category of the image to be recognized based on the first recognition result and the second recognition result includes: Determining whether the highest confidence level in the second recognition result is less than a second preset confidence threshold; When the highest confidence in the second recognition result is less than the second preset confidence threshold, summing the confidences of the same category in the first recognition result and the second recognition result to obtain a combined confidence for each category; The category with the highest combined confidence is used as the category of the image to be identified.

9. The method according to claim 8, characterized in that Also includes: When the highest confidence in the second recognition result is greater than or equal to the second preset confidence threshold, the category corresponding to the highest confidence in the second recognition result is used as the category of the image to be recognized.

10. The method according to claim 1, characterized in that Before acquiring the image to be identified, the terminal reasoning time, the server reasoning time and the information transmission time, the method further includes: Get a sample image; Inputting the sample image into an image recognition model, and recording the terminal inference time when the image recognition model recognizes the sample image; sending the sample image to the server; A recognition response fed back by the server is received, where the recognition response includes a server reasoning time for the server to recognize the sample image.

Images