Image recognition method, electronic device, storage medium, program product, and chip

Abstract

The application relates to the technical field of terminals, in particular to an image recognition method, an electronic device, a storage medium, a program product and a chip. In the method, the electronic device can acquire a first image to be processed, and perform multi-scale down-sampling on the first image to obtain a plurality of down-sampled images. Then, the electronic device can determine a plurality of target image blocks from the plurality of down-sampled images, and recognize image information in the first image according to the plurality of target image blocks. The target image blocks include valid information of the first image. In this way, the plurality of down-sampled images are obtained by performing multi-scale down-sampling on the first image, the target image blocks containing valid information are determined from the plurality of down-sampled images, the image information in the first image is recognized, the image blocks containing invalid information do not need to be subjected to image recognition, and the computing resources of the electronic device can be reduced.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image recognition method, electronic device, storage medium, software product, and chip. Background Technology

[0002] Image recognition technology is widely used in various fields such as medical image analysis, autonomous driving, security monitoring, and social media content analysis. Accurate image recognition capabilities can improve operational efficiency and security levels in these fields. In some image recognition methods, electronic devices can divide an entire image into multiple image blocks and analyze the information in each block to identify the overall image information. However, dividing the entire image into multiple blocks may include image blocks that do not contain valid semantic information (i.e., invalid information), such as background image blocks. Therefore, analyzing the entire image containing invalid information wastes the computing resources of electronic devices. Summary of the Invention

[0003] To address the problem that image recognition methods waste the computing resources of electronic devices, embodiments of this application provide an image recognition method, an electronic device, a storage medium, a program product, and a chip, including:

[0004] In a first aspect, embodiments of this application provide an image recognition method applied to an electronic device, comprising: acquiring a first image to be processed; performing multi-scale downsampling processing on the first image to obtain multiple downsampled images; determining multiple target image blocks from the multiple downsampled images, wherein the target image blocks include valid information in the first image; and recognizing image information in the first image based on the multiple target image blocks.

[0005] Based on the above scheme, multiple downsampled images are obtained by performing multi-scale downsampling processing on the first image, and target image blocks containing valid information are determined from the multiple downsampled images to identify the image information in the first image. This eliminates the need to perform image recognition on image blocks containing invalid information, thereby reducing the computing resources of electronic devices and improving the recognition efficiency of electronic devices.

[0006] In some implementations of the first aspect, determining multiple target image blocks from multiple downsampled images includes: performing edge detection and difference comparison processing on each downsampled image to obtain a local information map corresponding to each downsampled image; performing binarization processing on the local information map corresponding to each downsampled image to obtain a binary map corresponding to each downsampled image; the binary map corresponding to each downsampled image includes multiple sampled image blocks; and determining multiple target image blocks from the multiple downsampled images based on the multiple sampled image blocks in the binary map corresponding to each downsampled image.

[0007] In this embodiment, multiple downsampled images are obtained by performing multi-scale downsampling processing on the first image, and a target image block is determined from the multiple downsampled images. Compared with the method of dividing the first image into multiple image blocks and determining the target image block from them, the downsampled images obtained by the technical solution of this embodiment are smaller in size and contain fewer pixels. Therefore, the amount of data processing of the electronic device can be reduced in the process of determining the target image block from multiple downsampled images, thereby further reducing the computing resources of the electronic device and further improving the processing efficiency of the electronic device.

[0008] In some implementations of the first aspect, edge detection and difference comparison processing are performed on each of the multiple downsampled images to obtain a local information map corresponding to each downsampled image, including: determining the local information map corresponding to each downsampled image based on the edge information and neighborhood information corresponding to each downsampled image; wherein, the edge information is obtained by performing edge detection processing on each downsampled image, and the neighborhood information is obtained by performing difference comparison processing on each downsampled image.

[0009] It is understandable that edge information can be, for example, the outlines of image elements such as buttons, text blocks, and application icons in the corresponding image of the interface. Furthermore, strong edge information in each downsampled image can be represented using the first grayscale image corresponding to each downsampled image. The higher the pixel value of a pixel in the first grayscale image corresponding to each downsampled image, the higher the probability that the pixel represents the outline of an image element; that is, the more information the pixel contains.

[0010] The richness of information within the neighborhood of each pixel in each downsampled image can be represented by the second grayscale image corresponding to each downsampled image. The higher the pixel value of a pixel in the grayscale image corresponding to each downsampled image, the richer the information within that pixel's neighborhood, that is, the more information there is in that pixel's neighborhood.

[0011] In this embodiment, edge detection and difference comparison processing are performed on each downsampled image to determine the pixels with more information and the richness of information in the neighborhood of the pixel (the higher the richness, the more information in the neighborhood of the pixel). Subsequently, image information in the first image can be identified based on the image blocks composed of these pixels with more information. Compared with the method of identifying image information based on the whole image including invalid information, the computing resources of electronic devices can be further reduced and the processing efficiency of electronic devices can be improved.

[0012] In some implementations of the first aspect, the neighborhood information corresponding to each downsampled image is determined by: determining the variance of the pixel value of each pixel and its associated pixels in each downsampled image; and determining the neighborhood information corresponding to each downsampled image based on the variance of the pixel value of each pixel and its associated pixels.

[0013] For example, electronic device 100 can obtain such as Figure 8 The image shows the pixel value of pixel p1ij in the first grayscale image corresponding to the downsampled image I1, and the pixel values ​​of a 7×7 pixel array centered on pixel p1ij (as shown by the dashed lines in the figure). Furthermore, the electronic device 100 can determine the pixel value of pixel p1ij in the second grayscale image corresponding to the downsampled image I1 by using the variance of the pixel value of pixel p1ij and the pixel values ​​of the 7×7 pixel array.

[0014] In some implementations of the first aspect, a local information map corresponding to each downsampled image is determined based on the edge information and neighborhood information corresponding to each downsampled image, including: performing weighted fusion processing on the edge information and neighborhood information corresponding to each downsampled image to obtain the local information map corresponding to each downsampled image.

[0015] In some implementations of the first aspect, binarizing the local information map corresponding to each downsampled image to obtain a binary image corresponding to each downsampled image includes: binarizing the local information map corresponding to each downsampled image according to a preset pixel threshold to obtain a binary image corresponding to each downsampled image.

[0016] In this embodiment, by binarizing the local information map corresponding to the downsampled image, pixels containing valid information and pixels containing invalid information can be determined. Subsequently, image information in the first image can be identified based on the image block composed of pixels containing valid information as the target image block. Compared with the method of identifying image information based on the whole image including invalid information, the computing resources of electronic devices can be further reduced and the processing efficiency of electronic devices can be further improved.

[0017] In some implementations of the first aspect, the local information map corresponding to each downsampled image is binarized according to a preset pixel threshold to obtain a binary map corresponding to each downsampled image, including: determining the pixel values ​​in each downsampled image that are greater than the pixel threshold as a first value, and determining the pixel values ​​in each downsampled image that are less than or equal to the pixel threshold as a second value, to obtain a binary map corresponding to each downsampled image.

[0018] In some implementations of the first aspect, the multiple downsampled images include a second image, the size of which is smaller than that of the first image. Based on multiple sampled image blocks in the binary image corresponding to each downsampled image, multiple target image blocks are determined from the multiple downsampled images, including: if the first sampled image block in the binary image corresponding to the second image satisfies a first preset condition, the first image block corresponding to the first sampled image block in the second image is determined as the target image block; wherein, the first preset condition is that the number of pixels with a pixel value of a first value is greater than a number threshold.

[0019] In some implementations of the first aspect, the multiple downsampled images include a second image and a third image, wherein the size of the second image is smaller than the size of the first image, and the size of the third image is smaller than the size of the second image. Based on multiple sampled image blocks in the binary image corresponding to each downsampled image, multiple target image blocks are determined from the multiple downsampled images, including: if the second sampled image block in the binary image corresponding to the second image does not satisfy a first preset condition, and the third sampled image corresponding to the second sampled image in the binary image corresponding to the third image satisfies the first preset condition, then the downsampled image block corresponding to the third sampled image block in the third image is determined as the target image block; wherein, the first preset condition is that the number of pixels with a pixel value of a first value is greater than a number threshold.

[0020] In some implementations of the first aspect, identifying image information in a first image based on multiple target image blocks includes: identifying image information in a first image based on the position information, size information, and image feature information of each target image block in the multiple target image blocks; wherein, the position information is determined based on the position of each target image block in the first coordinate system corresponding to the first image, the size information is determined based on the size of the downsampled image to which each target image block belongs, and the image feature information is obtained by feature extraction of each target image block.

[0021] In some implementations, the electronic device can establish a first coordinate system with the lower left corner of the first image as the origin, a first direction X (horizontally to the right parallel to the paper) as the first direction, and a second direction Y (vertically upward parallel to the paper) as the second direction. Furthermore, the electronic device 100 can determine the position information of the target image block based on the position of its center in the first image (e.g., the position indicated by the dot in the figure) within the first coordinate system. Then, the electronic device can perform position encoding processing on the position information of each target image block based on a visual language model to obtain a position encoding vector corresponding to each target image block.

[0022] The electronic device can use the size of the downsampled image to which the target image patch belongs as the size of the target image patch, thus obtaining the size information of the target image patch. Then, the electronic device can perform size encoding processing on the size information corresponding to each target image patch to obtain the size encoding vector corresponding to each target image patch.

[0023] Furthermore, the electronic device can extract image feature information of the target image patch based on the ViT model. Moreover, the electronic device 100 can utilize the shared linear layers in the ViT model to convert the image feature information of the target image patch into a fixed-dimensional image feature vector.

[0024] In some implementations of the first aspect, the image recognition method further includes: receiving a first instruction from a user for a first image; and performing a first operation corresponding to the first instruction based on the image information of the first image.

[0025] For example, when a user issues a command to "describe the content of this page", the electronic device can generate a text description such as "the price is 299".

[0026] In a second aspect, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the first aspect and any possible implementation of the image recognition method provided in the first aspect.

[0027] Thirdly, embodiments of this application provide a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the first aspect and any possible implementation of the provided image recognition method.

[0028] Fourthly, embodiments of this application provide a computer program product, the computer program product including computer instructions, which, when executed by an electronic device, enable the electronic device to perform the image recognition method provided as described in the first aspect and any possible implementation of the first aspect.

[0029] Fifthly, embodiments of this application provide a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the first aspect and any possible implementation of the image recognition method provided in the first aspect.

[0030] It is understood that the specific implementation methods and beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions of any image recognition method in the first aspect or the various implementation methods provided in the first aspect, and will not be repeated here. Attached Figure Description

[0031] Figure 1 According to some embodiments of this application, a schematic diagram of an application scenario is shown;

[0032] Figure 2 According to some embodiments of this application, a flowchart of an image recognition method is shown;

[0033] Figure 3 According to some embodiments of this application, a schematic diagram is shown of a method for downsampling a first image to obtain multiple downsampled images;

[0034] Figure 4A According to some embodiments of this application, a schematic diagram of the correspondence between image blocks in a downsampled image I2 and a first image I1 is shown;

[0035] Figure 4B According to some embodiments of this application, a schematic diagram of the correspondence between downsampled image I4, downsampled image I2 and first image I1 is shown;

[0036] Figure 4C According to some embodiments of this application, a schematic diagram of the correspondence between downsampled image I8, downsampled image I4, downsampled image I2 and first image I1 is shown.

[0037] Figure 5 According to some embodiments of this application, a schematic diagram of determining the location information of a target image block is shown;

[0038] Figure 6 According to some embodiments of this application, a schematic diagram of determining the position information of a target image block is shown;

[0039] Figure 7 According to some embodiments of this application, a schematic diagram of determining edge information corresponding to a downsampled image is shown;

[0040] Figure 8 According to some embodiments of this application, a schematic diagram of determining a pixel neighborhood is shown;

[0041] Figure 9 According to some embodiments of this application, a schematic diagram of binary images corresponding to a first image I1, a downsampled image I2, a downsampled image I4, and a downsampled image I8 is shown.

[0042] Figure 10 A flowchart of an image recognition method is shown according to some embodiments of this application;

[0043] Figure 11 According to some embodiments of this application, a flowchart for determining a target image block is shown;

[0044] Figure 12 According to some embodiments of this application, a flowchart of a visual-text feature alignment method is shown;

[0045] Figure 13 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device 100 is shown;

[0046] Figure 14 According to some embodiments of this application, a schematic diagram of the software structure of an electronic device 100 is shown. Detailed Implementation

[0047] The embodiments of this application include, but are not limited to, image recognition methods, electronic devices, storage media, program products, and chips.

[0048] It is understood that the image recognition method mentioned in the embodiments of this application can be applied to electronic devices. These electronic devices can be referred to as terminals, user equipment (UE), mobile stations (MS), or mobile terminals (MT), etc.

[0049] In some implementations, electronic devices can be smartphones, smart TVs, smartwatches, smart bracelets, desktop computers, laptops, personal computers (PCs), virtual reality (VR) devices, augmented reality (AR) devices, terminals in industrial control, terminals in self-driving, terminals in remote medical surgery, terminals in smart grids, terminals in transport safety (such as vehicle-mounted terminals), terminals in smart cities, terminals in smart homes, and other similar devices.

[0050] It is understood that the image recognition methods mentioned in the embodiments of this application can be applied to fields such as intelligent voice interaction, medical image analysis, autonomous driving, security monitoring, and social media content analysis, and the embodiments of this application do not impose specific limitations.

[0051] like Figure 1The diagram illustrates an application scenario. In this scenario, electronic device 100 can display interface 110. When a user inputs a task into electronic device 100 via voice or text, electronic device 100 can take a screenshot of interface 110 to obtain an image corresponding to interface 110. Then, electronic device 100 can perform image recognition processing on the image corresponding to interface 110 to identify image information in the image. This includes image elements such as buttons, text boxes, and application icons, as well as information about the layout, function, and status of these image elements. Therefore, electronic device 100 can execute the user-input task based on the image information in the image corresponding to interface 110.

[0052] For example, continue as Figure 1 As shown, when the electronic device 100 detects a user's instruction to "order me a coffee," it can... Figure 1 Screenshotting interface 110 as shown in (a) yields the following result: Figure 1 The image corresponding to interface 110 shown in (b). Then, the electronic device 100 can divide the image corresponding to interface 110 into, as shown in (b). Figure 1 The system uses multiple image blocks (as shown by the dotted lines in the figure) as shown in (c), and performs image recognition processing on each of the multiple image blocks to determine the image information in the image corresponding to the interface 110. For example, it recognizes the shopping app icon. Then, the electronic device 100 can open the shopping app and repeat the above process of taking screenshots, dividing image blocks, and image recognition to order a cup of coffee that the user usually orders.

[0053] However, because some images in multiple image patches contain invalid information, for example, such as Figure 1 Image block B0 shown in (c) includes information about the background region, which does not include edge information of image elements; that is, the information about the background region is irrelevant to the image recognition information. Therefore, dividing the image corresponding to interface 110 into multiple image blocks and performing image recognition on each of these blocks would waste the computing resources of electronic device 100.

[0054] To address the aforementioned problems, this application provides an image recognition method. In this method, an electronic device 100 acquires a first image to be processed (e.g., the image corresponding to interface 110 mentioned above), performs multi-scale downsampling on the first image to obtain multiple downsampled images. Then, the electronic device 100 can determine multiple target image blocks from the multiple downsampled images and, based on these target image blocks, recognize image information in the first image. The target image blocks include valid information from the first image, such as edge information of image elements.

[0055] In this way, by performing multi-scale downsampling processing on the first image to obtain multiple downsampled images, and determining the target image block containing valid information from the multiple downsampled images, the image information in the first image can be identified. There is no need to perform image recognition on the image block containing invalid information, which can reduce the computing resources of the electronic device 100 and improve the recognition efficiency of the electronic device 100.

[0056] The image recognition method mentioned in the embodiments of this application will be described below. For example... Figure 2 The diagram shows a flowchart of an image recognition method that can be executed by an electronic device 100, such as the smartphone mentioned above.

[0057] For example, the image recognition method may include:

[0058] S201: Obtain the first image to be processed.

[0059] It is understood that the first image can be a screenshot of the interface 110 displayed by the electronic device 100, an image obtained from a gallery, or an image determined based on a multi-dimensional pixel array obtained from a server. This application embodiment does not impose any limitations.

[0060] It is understandable that after acquiring the first image, the electronic device 100 can perform image preprocessing on the first image to obtain a preprocessed first image. For example, the electronic device 100 can perform processing on the first image such as resizing, normalization, noise reduction, and image enhancement.

[0061] In some implementations, the electronic device 100 can resize the first image. For example, if the first image is a non-standard size, the electronic device 100 can resize it to a standard size. The standard size could be 1080×1920.

[0062] After resizing the first image, the electronic device 100 can normalize the pixel values ​​of each pixel in each channel of the first image. For example, the electronic device 100 can map the pixel values ​​of each pixel in each channel of the first image from the integer range of [0, 255] to the floating-point range of [0, 1], and then scale them to the numerical range of the overall distribution using [0.485, 0.456, 0.406] as the mean and [0.299, 0.224, 0.225] as the variance.

[0063] It is understood that the electronic device 100 may also scale the pixel values ​​of each pixel in each channel of the first image based on other means or variances, and this application embodiment does not make specific limitations.

[0064] In this embodiment of the application, by preprocessing the first image, the difference between pixel values ​​of each pixel in each channel in the first image can be reduced.

[0065] S202: Perform multi-scale downsampling processing on the first image to obtain multiple downsampled images.

[0066] It is understood that after acquiring the first image or preprocessing the first image, the electronic device 100 can perform multi-scale downsampling processing on the first image or the preprocessed first image (hereinafter collectively referred to as the first image) to obtain multiple downsampled images In of different sizes. In some implementations, the electronic device 100 can perform multi-scale downsampling processing on the first image using algorithms such as bilinear interpolation, average pooling, or max pooling.

[0067] For example, such as Figure 3 As shown, when the size of the first image I1 is 1080×1920, the electronic device 100 can perform a 1x downsampling process on the first image I1 to obtain a downsampled image I1 of 1080×1920 (i.e., the first image I1). The electronic device 100 can perform a 2x downsampling process on the first image I1 to obtain a downsampled image I2 of 540×960. The electronic device 100 can perform a 4x downsampling process on the first image I1 to obtain a downsampled image I4 of 270×480. The electronic device 100 can perform an 8x downsampling process on the first image I1 to obtain a downsampled image I8 of 135×240. Thus, the size of the downsampled image I1 can be 1, the size of the downsampled image I2 can be 1 / 2, the size of the downsampled image I4 can be 1 / 4, and the size of the downsampled image I8 can be 1 / 8.

[0068] It is understandable that in downsampled images I1, I2, I4, and I8, one pixel in one image can correspond to one or more pixels in another image. For example, the pixel information of one pixel in a small image can represent the pixel information of multiple pixels in a large image. In some implementations, the electronic device 100 can define multiple pixels as an image block. Thus, the pixel information of one pixel in a small image can represent the pixel information of an image block in a large image.

[0069] For example, such as Figure 4A As shown, the pixel information of pixel p211 in the downsampled image I2 can be represented as the pixel information of image block B1 (the image block composed of pixels p111, p112, p121 and p122) in the downsampled image I1.

[0070] like Figure 4BAs shown, the pixel information of pixel p411 in the downsampled image I4 can represent the pixel information of image block B2 (the image block composed of pixels p111, p112, p113, p114, p121, p122, p123, p124, p131, p132, p133, p134, p141, p142, p143, and p144) in the downsampled image I1.

[0071] The pixel information of pixel p411 in the downsampled image I4 can also be represented as the pixel information of image block B3 (the image block composed of pixels p211, p212, p221 and p222) in the downsampled image I2.

[0072] like Figure 4C As shown, the pixel information of pixel p811 in the downsampled image I8 can represent the pixel information of image block B4 (the image block composed of pixels p111...p118, p121...p128, p131...p138, p141...p148, p151...p158, p161...p168, p171...p178, p181...p188) in the downsampled image I1.

[0073] The pixel information of pixel p811 in the downsampled image I8 can also represent the pixel information of image block B5 (the image block composed of pixels p211, p212, p213, p214, p221, p222, p223, p224, p231, p232, p233, p234, p241, p242, p243, and p244) in the downsampled image I2.

[0074] The pixel information of pixel p811 in the downsampled image I8 can also be represented as the pixel information of image block B6 (the image block composed of pixels p411, p412, p421 and p422) in the downsampled image I4.

[0075] S203: Determine multiple target image blocks from multiple downsampled images, wherein the target image blocks include valid information from the first image.

[0076] It is understandable that after obtaining multiple downsampled images, for each downsampled image In, the electronic device 100 can estimate the local information of each downsampled image In based on edge detection and difference comparison algorithms to quantify the information complexity of different image blocks in each downsampled image. The local information of each downsampled image In can be represented by a corresponding local information map Cn. This local information map Cn can be represented using a grayscale image. In the local information map corresponding to each downsampled image In, the higher the pixel value, the more information is contained within that pixel and its neighborhood (e.g., the region consisting of N×N pixels centered on each pixel).

[0077] After obtaining the local information maps corresponding to multiple downsampled images In, the electronic device 100 can compare the pixel value of each pixel in the local information map Cn corresponding to each downsampled image In with the pixel threshold T, thereby determining the binary map Cn' corresponding to each downsampled image In. For example, if the pixel value of a certain pixel is greater than the pixel threshold T, the pixel value of the corresponding pixel in the binary map Cn' corresponding to the downsampled image In is set to a first value of 1, indicating that the pixel in the downsampled image In is a high-information pixel, that is, a pixel containing valid information; if the pixel value of a certain pixel is less than or equal to the pixel threshold T, the pixel value of the corresponding pixel in the binary map Cn' corresponding to the downsampled image In is set to a second value of 0, indicating that the pixel in the downsampled image In is a low-information pixel, that is, a pixel containing invalid information.

[0078] After determining the binary image Cn' corresponding to each downsampled image In, the electronic device 100 can divide the binary image Cn' corresponding to each downsampled image In into multiple sampled image blocks based on a preset size. Then, the electronic device 100 can determine whether each sampled image block in the binary image Cn' corresponding to each downsampled image In meets a preset condition. If the sampled image block meets the preset condition, the image block corresponding to the downsampled image In is determined as the target image block. The preset condition can be that the number of pixels with a pixel value of a first value is greater than a number threshold.

[0079] In this embodiment of the application, multiple downsampled images are obtained by performing multi-scale downsampling processing on the first image, and a target image block is determined from the multiple downsampled images. Compared with the method of dividing the first image into multiple image blocks and determining the target image block from them, the downsampled images obtained by the technical solution of this embodiment of the application are smaller in size and contain fewer pixels. Therefore, the amount of data processing of the electronic device 100 can be reduced in the process of determining the target image block from the multiple downsampled images, thereby further reducing the computing resources of the electronic device 100 and further improving the processing efficiency of the electronic device 100.

[0080] S204: Identify image information in the first image based on multiple target image blocks.

[0081] In some implementations, the electronic device 100 can determine the position information of each target image block based on its position in the first coordinate system corresponding to the first image. Furthermore, the electronic device 100 can determine the size information of each target image block based on the size of the downsampled image to which it belongs. Additionally, the electronic device 100 can perform feature extraction processing on each target image block to obtain image feature information for each target image block.

[0082] Furthermore, the electronic device 100 can identify image information in the first image based on the position information, size information, and image feature information of each target image block among multiple target image blocks. For example, image elements such as buttons, text boxes, and application icons, as well as information such as the layout, function, and status of these image elements.

[0083] like Figure 5 The diagram illustrates a flowchart for identifying image information in a first image. In some implementations, the electronic device 100 can determine the image information in the first image based on a vision language model (VLM), the positional information of each target image patch, the size information of each target image patch, and the image feature information of each target image patch. That is, multiple target image patches are input into the VLM model, and the model outputs information about the understanding of the content of the first image.

[0084] In some methods for determining the location information of a target image patch, such as Figure 6 As shown, the electronic device 100 can establish a first coordinate system with the lower left corner of the first image as the origin, a first direction X (horizontally to the right parallel to the paper), and a second direction Y (vertically upward parallel to the paper). Furthermore, the electronic device 100 can determine the position information of the target image block based on the position of its center in the first image I1 (e.g., the position indicated by the dots in the figure) within the first coordinate system. Then, the electronic device 100 can perform position encoding processing on the position information of each target image block based on a large language model (e.g., a visual language model) to obtain a position encoding vector corresponding to each target image block. For example, the electronic device 100 can encode the position information of each target image block into a high-dimensional vector based on a multilayer perceptron (MLP) in the visual language model.

[0085] In some methods of determining the size information of a target image patch, the electronic device 100 can use the size of the downsampled image to which the target image patch belongs as the size of the target image patch to obtain the size information of the target image patch. For example, if the target image patch belongs to downsampled image I1, the size of the target image patch can be 1; if the target image patch belongs to downsampled image I2, the size of the target image patch can be 1 / 2; if the target image patch belongs to downsampled image I4, the size of the target image patch can be 1 / 4; and if the target image patch belongs to downsampled image I8, the size of the target image patch can be 1 / 8. Then, the electronic device 100 can perform size encoding processing on the size information corresponding to each target image patch to obtain a size encoding vector corresponding to each target image patch.

[0086] In some methods of determining image feature information of a target image patch, electronic device 100 can extract image feature information of the target image patch based on the ViT model (a computer vision model based on the Transform architecture) in a visual language model. For example, electronic device 100 can extract image feature information of the target image patch (corresponding to...) based on the ViT model in a visual language model. Figure 5 Image block pixels) are linearly projected onto the image block (corresponding to) Figure 5 In image patch linear projection, for example, the shared linear projection layer in the ViT model can be used to convert the image feature information of the target image patch into a fixed-dimensional (e.g., 768-dimensional) image feature vector. This image feature vector can also be called patch embedding.

[0087] For example, if the position information of the target image patch is (x, y), the size information is s, and the dimension of the image feature information is d, then the position embedding of the target image patch can be a vector of dimension [d, 4].

[0088] Thus, in d-dimensional features, even-numbered terms can be encoded using a sine function, with the specific encoding method as follows:

[0089] PE(x,y,s,2k)=[sin((xs) / 10000 （2k / d） ), sin((x+s) / 10000 （2k / d） ), sin((ys) / 10000 （2k / d） sin((y+s) / 10000) （2k / d） ))];

[0090] Where k takes values ​​in the interval [1, d / / 2].

[0091] In d-dimensional features, odd-numbered terms can be encoded using a cosine function, as follows:

[0092] PE(x,y,s,2k+1)=[cos((xs) / 10000 （2k / d） ), cos ((x+s) / 10000 (2k / d)), cos ((ys) / 10000 （2k / d） cos((y+s) / 10000) （2k / d） ))];

[0093] Where k takes values ​​in the interval [1, d / / 2].

[0094] After determining the position encoding vector, size encoding vector, and image feature information corresponding to each target image block, the electronic device 100 can fuse (e.g., add or concatenate) the position encoding vector and size encoding vector corresponding to each target image block to obtain the comprehensive encoding vector (corresponding to) each target image block. Figure 5 The image block position and size are determined, and the comprehensive encoding vector corresponding to each target image block is added to the image block embedding corresponding to the target image block as a visual feature vector representing the unique identity of the target image block.

[0095] After determining the visual feature vector of each target image patch, the electronic device 100 can input the visual feature vector of each target image patch into the Transform encoder. Utilizing the multi-head self-attention network in each layer of the Transform encoder, multiple "heads" perform parallel computation to capture the correlation between target image patches from different angles, thereby determining the correlation (corresponding to) between each target image patch and other target image patches. Figure 5 (Feature aggregation between image patches in the image). Furthermore, the visual feature vector of each target image patch is non-linearly transformed using the feedforward neural network in each layer of the Transform encoder (corresponding to...). Figure 5 (Intra-image patch feature transformation) to enhance the expressive power of the visual feature vector of each target image patch.

[0096] Next, the electronic device 100 can utilize multiple linear layers to align the output of the Transform encoder with the text feature space generated in response to the user's instructions for the first image (corresponding to...). Figure 5 (Visual-text feature alignment) is used to determine the image information of the first image.

[0097] In this embodiment, multiple downsampled images are obtained by performing multi-scale downsampling processing on the first image, and target image blocks containing valid information are determined from the multiple downsampled images to identify the image information in the first image. This eliminates the need to perform image recognition on image blocks containing invalid information, thereby reducing the computing resources of the electronic device 100 and improving the recognition efficiency of the electronic device 100.

[0098] The following section describes the specific implementation method for determining the local information map Cn corresponding to each downsampled image In.

[0099] In some implementations, the electronic device 100 can perform edge detection and difference comparison processing on each downsampled image In among multiple downsampled images to obtain a local information map corresponding to each downsampled image In.

[0100] For example, the electronic device 100 can perform edge detection on each downsampled image In to obtain the edge information corresponding to each downsampled image In.

[0101] That is, the electronic device 100 can employ an edge detection algorithm to determine the strong edge information in each downsampled image In. The strong edge information can be, for example, the outlines of image elements such as buttons, text blocks, and application icons in the image corresponding to the interface 110. Furthermore, the strong edge information in each downsampled image In can be represented using a first grayscale image corresponding to each downsampled image In. The higher the pixel value of a pixel in the first grayscale image corresponding to each downsampled image In, the higher the probability that the pixel represents the outline of an image element; that is, the more information the pixel contains.

[0102] For example, for Figure 1 The electronic device 100 can use the Canny edge detection algorithm to detect the image corresponding to the interface shown in (b). Figure 7 The strong edge information of the downsampled image I1 shown in (b) is obtained, and the following is used: Figure 7 The first grayscale image shown in (b) represents the strong edge information in the downsampled image I1.

[0103] After determining the first grayscale image corresponding to each downsampled image In, the electronic device 100 can perform difference comparison processing on each downsampled image In to obtain the neighborhood information corresponding to each downsampled image In. In some implementations, the electronic device 100 can determine the pixel value variance between each pixel and its associated pixels in each downsampled image In, and determine the neighborhood information corresponding to each downsampled image In based on the pixel value variance between each pixel and its associated pixels.

[0104] That is, the electronic device 100 can employ a difference comparison algorithm to quantify the richness of information in the neighborhood of each pixel in each downsampled image In. The richness of information in the neighborhood of each pixel in each downsampled image In can be represented by the second grayscale image corresponding to each downsampled image In. The higher the pixel value of a pixel in the grayscale image corresponding to each downsampled image In, the higher the richness of information in the neighborhood of that pixel, that is, the more information is contained in the neighborhood of that pixel.

[0105] For example, electronic device 100 can obtain such as Figure 8 The image shows the pixel value of pixel p1ij in the first grayscale image corresponding to the downsampled image I1, and the pixel values ​​of a 7×7 pixel array centered on pixel p1ij (as shown by the dashed lines in the figure). Furthermore, the electronic device 100 can determine the pixel value of pixel p1ij in the second grayscale image corresponding to the downsampled image I1 by using the variance of the pixel value of pixel p1ij and the pixel values ​​of the 7×7 pixel array.

[0106] After determining the edge information and neighborhood information corresponding to each downsampled image In, the electronic device 100 can determine the local information map corresponding to each downsampled image In based on the edge information and neighborhood information corresponding to each downsampled image In. In some implementations, the electronic device 100 can perform weighted fusion processing on the edge information and neighborhood information corresponding to each downsampled image In to obtain the local information map corresponding to each downsampled image In.

[0107] For example, the electronic device 100 can perform weighted fusion of the pixel value of each pixel in the first grayscale image corresponding to each downsampled image In and the pixel value of the corresponding pixel in the second grayscale image corresponding to each downsampled image In to generate a local information map Cn corresponding to each downsampled image In.

[0108] In this embodiment, edge detection and difference comparison processing are performed on each downsampled image to determine the pixels with more information and the richness of information in the neighborhood of the pixel (the higher the richness, the more information in the neighborhood of the pixel). Subsequently, image information in the first image can be identified based on the image blocks composed of these pixels with more information. Compared with the method of identifying image information based on the whole image including invalid information, the computing resources of electronic devices can be further reduced and the processing efficiency of electronic devices can be improved.

[0109] The following section describes the specific implementation method for determining the binary image Cn' corresponding to each downsampled image In.

[0110] In some implementations, the electronic device 100 can binarize the local information map corresponding to each downsampled image In to obtain a binary map corresponding to each downsampled image In. The binary map corresponding to each downsampled image In may include multiple sampled image blocks.

[0111] For example, the electronic device 100 can binarize the local information map corresponding to each downsampled image according to a preset pixel threshold to obtain a binary map Cn' corresponding to each downsampled image In. For instance, the electronic device 100 can determine the pixel values ​​in the local information map Cn corresponding to each downsampled image In that are greater than the pixel threshold as a first value, and determine the pixel values ​​in the local information map Cn corresponding to each downsampled image In that are less than or equal to the pixel threshold as a second value, thereby obtaining the binary map Cn' corresponding to each downsampled image In.

[0112] Thus, for Figure 3 The electronic device 100 can generate downsampled images I1, I2, I4, and I8 as shown. Figure 9 Middle (a) to Figure 9 The binary images corresponding to each downsampled image In are shown in (d). For example, the binary image C1' corresponding to downsampled image I1, the binary image C2' corresponding to downsampled image I2, the binary image C4' corresponding to downsampled image I4, and the binary image C8' corresponding to downsampled image I8.

[0113] In this embodiment, by binarizing the local information map corresponding to the downsampled image, pixels containing valid information and pixels containing invalid information can be determined. Subsequently, image information in the first image can be identified based on image blocks composed of pixels containing valid information as target image blocks. Compared to identifying image information based on the entire image including invalid information, this method further reduces the computational resources of the electronic device and improves its processing efficiency. A specific implementation method for determining multiple target image blocks from multiple downsampled images In is described below.

[0114] In some implementations, the electronic device 100 can determine multiple target image blocks from multiple downsampled images In based on multiple sampled image blocks in the binary image Cn' corresponding to each downsampled image In.

[0115] For example, when multiple downsampled images include a second image, i.e., the size of the second image is smaller than the size of the first image, the electronic device 100 can determine whether each sampled image block in the binary image corresponding to the second image meets a preset condition. If the first sampled image block in the binary image corresponding to the second image meets the preset condition, the electronic device 100 can determine the first image block corresponding to the first sampled image block in the second image as the target image block.

[0116] In the case of multiple downsampled images including a second image and a third image, and the size of the third image is smaller than the size of the second image, if the second sampled image block in the binary image corresponding to the second image does not meet the preset conditions, but the third sampled image block corresponding to the second sampled image block in the binary image corresponding to the third image meets the preset conditions, the electronic device 100 can determine the second image block corresponding to the third sampled image block in the third image as the target image block.

[0117] For example, for Figure 3 The electronic device 100 determines the image block Bk from the binary image C8' corresponding to the downsampled image I1, downsampled image I2, downsampled image I4 and downsampled image I8 shown. Based on the pixel values ​​of each pixel in the image block Bk, it determines whether the corresponding image block in the downsampled image I8 is the target image block.

[0118] When all pixel values ​​in image block Bk are 0, electronic device 100 can determine that the corresponding image block in the first image still lacks information after 8x downsampling. Therefore, electronic device 100 can determine the corresponding image block in the downsampled image I8 as a non-target image block.

[0119] If there are pixels with non-zero values ​​in image block Bk, electronic device 100 can determine that the corresponding image block in the first image contains important information. Therefore, electronic device 100 can determine the image block corresponding to image block Bk in the binary image C4' corresponding to downsampled image I4, and determine whether the corresponding image block in downsampled image I4 is the target image block based on the pixel values ​​of the pixels in the image block corresponding to image block Bk in the binary image C4' corresponding to downsampled image I4.

[0120] When the pixel values ​​of all pixels in the image block corresponding to the image block Bk in the binary image C4' corresponding to the downsampled image I4 are 0, the electronic device 100 can determine the corresponding image block in the downsampled image I4 as a non-target image block.

[0121] In the binary image C4' corresponding to the downsampled image I4, there are pixels in the image block corresponding to image block Bk with non-zero pixel values. The electronic device 100 can determine the image block corresponding to image block Bk in the binary image C2' corresponding to the downsampled image I2, and determine whether the corresponding image block in the downsampled image I2 is the target image block based on the pixel values ​​of the pixels in the image block corresponding to image block Bk in the binary image C2' corresponding to the downsampled image I2.

[0122] When the pixel values ​​of the pixels in the image block corresponding to the image block Bk in the binary image C2' corresponding to the downsampled image I2 are all 0, the electronic device 100 can determine the corresponding image block in the downsampled image I2 as a non-target image block.

[0123] In the binary image C2' corresponding to the downsampled image I2, there are pixels in the image block corresponding to image block Bk with non-zero pixel values. The electronic device 100 can determine the image block corresponding to image block Bk in the binary image C1' corresponding to the downsampled image I1, and determine whether the corresponding image block in the downsampled image I1 is the target image block based on the pixel values ​​of the pixels in the image block corresponding to image block Bk in the binary image C1' corresponding to the downsampled image I1.

[0124] When the pixel values ​​of all pixels in the image block corresponding to the image block Bk in the binary image C4' corresponding to the downsampled image I1 are 0, the electronic device 100 can determine the corresponding image block in the downsampled image I1 as a non-target image block.

[0125] In the binary image C4' corresponding to the downsampled image I1, there are pixels in the image block corresponding to the image block Bk with non-zero pixel values. The electronic device 100 can determine the corresponding image block in the downsampled image I1 as the target image block.

[0126] like Figure 4C In the downsampled image I8 shown, pixel p811, Figure 9 In the binary image C8' corresponding to the downsampled image I8 shown in (d), there are pixels with non-zero values ​​in the corresponding image block; Figure 4C Image block B6 in the downsampled image I4 (i.e., the image block corresponding to pixel p811 in the downsampled image I8 in the downsampled image I4) is shown in Figure 9 In the binary image C4' corresponding to the downsampled image I4 shown in (c), there are pixels in the image block with non-zero pixel values; Figure 4C Image block B5 in the downsampled image I2 (i.e., the image block corresponding to pixel p811 in the downsampled image I8 in the downsampled image I2) is shown in Figure 9 In the binary image C2' corresponding to the downsampled image I2 shown in (b), there are pixels in the image block where the pixel value is non-zero; Figure 4CImage block B4 in the downsampled image I1 (i.e., the image block corresponding to pixel P811 in the downsampled image I8 in the downsampled image I1) is shown in Figure 9 In the image block corresponding to the binary image C1' shown in (a), there are pixels with non-zero values ​​in the image block. The electronic device 100 can determine the image block B4 in the downsampled image I1 as the target image block.

[0127] In this embodiment of the application, multiple downsampled images are obtained by performing multi-scale downsampling processing on the first image, and a target image block is determined from the multiple downsampled images. Compared with the method of dividing the first image into multiple image blocks and determining the target image block from them, the downsampled images obtained by the technical solution of this embodiment of the application are smaller in size and contain fewer pixels. Therefore, the amount of data processing of the electronic device 100 can be reduced in the process of determining the target image block from the multiple downsampled images, thereby further reducing the computing resources of the electronic device 100 and further improving the processing efficiency of the electronic device 100.

[0128] The following is combined with Figure 10 The flowchart of the image recognition method shown is as follows: Figure 11 The flowchart shown illustrates the process of determining the target image patch, and Figure 12 The flowchart illustrating the visual-text feature alignment method further describes the image recognition method mentioned in the embodiments of this application. It is understood that this image recognition method can be executed by the electronic device 100; exemplarily, the image recognition method may include:

[0129] S1001: Obtain the first image.

[0130] In some implementations, such as Figure 11 As shown, the electronic device 100 can acquire the first image I1. For example, the electronic device 100 can take a screenshot of the interface 110 displayed by the electronic device 100 to obtain the first image, the electronic device 100 can also read the first image from a gallery, or the electronic device 100 can read the multi-dimensional pixel array of the first image from a service to obtain the first image.

[0131] S1002: Preprocess the first image to obtain the preprocessed first image.

[0132] In some implementations, after acquiring the first image, the electronic device 100 can perform image preprocessing and multi-scale downsampling on the first image I1. The electronic device 100 can perform preprocessing on the first image, such as resizing and normalization, to obtain a preprocessed first image. The specific implementation of the preprocessing of the first image can be found in S201, and will not be elaborated here.

[0133] S1003: Perform multi-scale downsampling on the preprocessed first image, and perform local information estimation on the multi-frame downsampled images obtained by multi-scale downsampling to obtain the local information map corresponding to each frame downsampled image.

[0134] In some implementations, the electronic device 100 can perform multi-scale downsampling processing on the preprocessed first image to obtain multiple frames of downsampled images, for example... Figure 11 The generated downsampled images I2, I4, and I8 are then processed. The electronic device 100 can then estimate the local information of each downsampled image In based on an edge detection algorithm and a difference comparison algorithm, obtaining a local information map corresponding to each downsampled image In, for example... Figure 11 The local information maps C1, C2, C4, and C8 corresponding to the downsampled image are determined in the diagram. In each local information map corresponding to the downsampled image In, the higher the pixel value of a pixel, the more information is contained within that pixel and its neighborhood.

[0135] S1004: Based on the local information map corresponding to each frame of downsampled image, dynamically divide and discard multiple frames of downsampled images to obtain multiple target image blocks.

[0136] In some implementations, the electronic device 100 can determine the binary image corresponding to each downsampled image In based on the local information map corresponding to each downsampled image In, for example, Figure 11 The binary images C1', C2', C4', and C8' corresponding to the downsampled image are determined in the diagram. In each binary image corresponding to the downsampled image In, pixels with a value of 1 are high-information pixels, i.e., pixels containing valid information; pixels with a value of 0 are low-information pixels, i.e., pixels containing invalid information.

[0137] Next, the electronic device 100 can divide the binary image corresponding to the downsampled image In into multiple sampled image blocks based on a preset size, and determine the image blocks in the corresponding downsampled image where the number of sampled image blocks with a pixel value of 1 is greater than a threshold as target image blocks, and discard the image blocks in the corresponding downsampled image where the number of sampled image blocks with a pixel value of 0 is greater than a threshold, so as to realize the dynamic segmentation and discarding of image blocks in the downsampled image.

[0138] For example, such as Figure 11As shown, the electronic device 100 can perform hierarchical decision-making on multiple downsampled images starting from C8'. It determines whether all pixel values ​​of a sampled image block in C8' are 0. If so, it locks the block and uses the corresponding image block in I8. If not, it determines whether all pixel values ​​of the sampled image block in C4' are 0. If so, it locks the block and uses the corresponding image block in I4. If not, it determines whether all pixel values ​​of the sampled image block in C2' are 0. If so, it locks the block and uses the corresponding image block in I2. If not, it determines whether all pixel values ​​of the sampled image block in C1' are 0. If so, it discards the block. If not, it uses the image block corresponding to I1, thus obtaining a dynamic image block sequence, i.e., multiple target image blocks.

[0139] S1005: Encode the image block position and size for each of the multiple target image blocks to obtain the position encoding vector and size encoding vector for each target image block.

[0140] In some implementations, after determining multiple target image blocks, the electronic device 100 can obtain the position information of each target image block by positioning the center of each target image block in the first coordinate system corresponding to the first image. Then, the electronic device 100 can perform position encoding on the position information of each target image block to obtain a position encoding vector corresponding to each target image block.

[0141] Furthermore, the electronic device 100 can use the size of the downsampled image to which the target image block belongs as the size of the target image block to obtain the size information of the target image block. Then, the electronic device 100 can perform size encoding processing on the size information corresponding to each target image block to obtain the size encoding vector corresponding to each target image block.

[0142] S1006: Extract image block features from each of the multiple target image blocks to obtain the image feature vector of each target image block.

[0143] In some implementations, the electronic device 100 can extract image feature information of the target image patch based on the ViT model.

[0144] S1007: Based on the position encoding vector, size encoding vector, image feature vector, and text model of multiple image blocks, the first image is analyzed to obtain the analysis results.

[0145] It is understandable that a text model can be a large language model (LLM).

[0146] like Figure 12The diagram illustrates a flowchart of an approach to understanding image content based on a large language model output. For example, when a user inputs a task into the electronic device 100 via voice or text, such as "Find me price information" or "Describe the content of this page," the electronic device 100 can determine the corresponding text prompt words based on the voice or text. Figure 12 The text prompts (in the text) can be used to guide the text model to focus its attention on a specific task.

[0147] After determining the text prompt words, the electronic device 100 can encode the text prompt words based on a text editor in a large language model (LLM) to obtain text embedding codes (corresponding to...). Figure 12 Text embedding encoding in (text).

[0148] Then, the electronic device 100 can perform feature concatenation processing on the visual feature vector and text embedding encoding, and based on the large language model (corresponding to...) Figure 12 The large language model in the image performs visual-language association and logical reasoning analysis on the concatenated feature sequence. For example, it associates specific regions (e.g., buttons) in the first image with specific words (e.g., click) in the prompt, and infers the purpose of image elements based on the layout and content of the first image.

[0149] S1008: Output understanding information about the content of the first image based on the analysis results.

[0150] In some implementations, after performing visual-linguistic association and logical reasoning analysis on the concatenated feature sequence based on a large language model, the large language model can generate a coherent and accurate textual description (corresponding to...) based on the analysis results. Figure 12 The output in the image contains information about the understanding of the image content. For example, "The price is 299 yuan".

[0151] It is understood that the image recognition method mentioned in the embodiments of this application can be applied to the electronic device 100. The electronic device 100 mentioned in the embodiments of this application will be described below. For example... Figure 13 The diagram shows a schematic of the hardware structure of an electronic device 100.

[0152] like Figure 13As shown, the electronic device 100 may include a processor 1310, an external memory interface 1320, an internal memory 1321, a universal serial bus (USB) interface 1330, a charging management module 1340, a power management module 1341, a battery 1342, an antenna 1, an antenna 2, a mobile communication module 1350, a wireless communication module 1360, an audio module 1370, a speaker 1370A, a receiver 1370B, a microphone 1370C, a headphone jack 1370D, a sensor module 1380, buttons 1390, a motor 1391, an indicator 1392, a camera 1393, a display screen 1394, and a subscriber identification module (SIM) card interface 1395, etc. The sensor module 1380 may include a pressure sensor 1380A, a gyroscope sensor 1380B, a barometric pressure sensor 1380C, a magnetic sensor 1380D, an accelerometer sensor 1380E, a distance sensor 1380F, a proximity sensor 1380G, a fingerprint sensor 1380H, a temperature sensor 1380J, a touch sensor 1380K, an ambient light sensor 1380L, a bone conduction sensor 1380M, etc.

[0153] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0154] Processor 1310 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0155] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0156] The processor 1310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1310 is a cache memory. This memory can store instructions or data that the processor 1310 has just used or that are used repeatedly. If the processor 1310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1310, and thus improves the efficiency of the system.

[0157] In some implementations, the processor 1310 can acquire a first image to be processed (e.g., the image corresponding to interface 110 mentioned above), and perform multi-scale downsampling on the first image to obtain multiple downsampled images. Then, the processor 1310 can determine multiple target image blocks from the multiple downsampled images, and identify image information in the first image based on the multiple target image blocks. The target image blocks include valid information from the first image.

[0158] In this way, by performing multi-scale downsampling processing on the first image to obtain multiple downsampled images, the target image block containing valid information can be determined from the multiple downsampled images to identify the image information in the first image. There is no need to perform image recognition on the image block containing invalid information, which can reduce the computing resources of the electronic device 100 and improve the processing efficiency of the electronic device 100.

[0159] In some embodiments, the processor 1310 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0160] The I2C interface is a bidirectional synchronous serial bus, consisting of a serial data line (SDA) and a serial clock line (SCL). The I2S interface can be used for audio communication. The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. The UART interface is a universal serial data bus used for asynchronous communication. The MIPI interface can be used to connect the processor 1310 to peripheral devices such as the display 1394 and the camera 1393. The GPIO interface is configurable via software. GPIO interfaces can be configured as control signals or data signals. The USB interface 1330 is a USB standard compliant interface, specifically a Mini USB interface, a Micro USB interface, or a USB Type-C interface.

[0161] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0162] The charging management module 1340 is used to receive charging input from the charger. The power management module 1341 is used to connect the battery 1342, and the charging management module 1340 is connected to the processor 1310.

[0163] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 1350, wireless communication module 1360, modem processor and baseband processor, etc.

[0164] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1350 can provide solutions for wireless communication applications, including 2G / 3G / 4G / 5G, on electronic devices 100.

[0165] A modem processor may include a modulator and a demodulator.

[0166] The wireless communication module 1360 can provide solutions for wireless communication applications on electronic devices 100, including UWB, wireless local area networks (WLAN) (such as wireless fidelity (WiFi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0167] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 1350, and antenna 2 is coupled to wireless communication module 1360, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0168] Electronic device 100 implements display functions through GPU, display screen 1394, and application processor.

[0169] Display screen 1394 is used to display images, videos, etc.

[0170] Electronic device 100 can perform shooting functions through ISP, camera 1393, video codec, GPU, display 1394 and application processor.

[0171] The ISP (Internet Service Provider) processes data fed back from the camera 1393. The camera 1393 is used to capture still images or video. The digital signal processor (DSP) processes digital signals; in addition to digital image signals, it can process other digital signals. The video codec is used to compress or decompress digital video. The NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own.

[0172] The external memory interface 1320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 1310 through the external memory interface 1320 to perform data storage functions.

[0173] Internal memory 1321 can be used to store computer executable program code, which includes instructions. Processor 1310 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 1321.

[0174] Electronic device 100 can implement audio functions, such as music playback and recording, through an audio module 1370, a speaker 1370A, a receiver 1370B, a microphone 1370C, a headphone jack 1370D, and an application processor.

[0175] Audio module 1370 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 1370 can also be used for encoding and decoding audio signals.

[0176] The speaker 1370A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The receiver 1370B, also known as a "handpiece," is used to convert audio electrical signals into sound signals. The microphone 1370C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. The headphone jack 1370D is used to connect wired headphones.

[0177] The pressure sensor 1380A senses pressure signals and converts them into electrical signals. The gyroscope sensor 1380B determines the motion posture of the electronic device 100. The barometric pressure sensor 1380C measures air pressure. The magnetic sensor 1380D includes a Hall effect sensor. The electronic device 100 can use the magnetic sensor 1380D to detect the opening and closing of the flip cover. The accelerometer sensor 1380E detects the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The distance sensor 1380F measures distance. The proximity sensor 1380G may include, for example, a light-emitting diode (LED) and a photodetector, such as a photodiode. The ambient light sensor 1380L senses ambient light intensity. The fingerprint sensor 1380H collects fingerprints. The temperature sensor 1380J detects temperature. The touch sensor 1380K is also called a "touch panel." The bone conduction sensor 1380M acquires vibration signals.

[0178] Buttons 1390 include a power button, volume buttons, etc. A motor 1391 can generate vibration alerts. An indicator 1392 can be an indicator light, used to indicate charging status, battery level changes, and also to indicate messages, missed calls, notifications, etc. A SIM card interface 1395 is used to connect a SIM card.

[0179] like Figure 14 The diagram shows a schematic representation of the software structure of an electronic device 100.

[0180] Layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the hardware abstraction layer (HAL). Figure 14 (Not illustrated in the text), and the kernel layer.

[0181] The application layer can include a series of application packages.

[0182] like Figure 14 As shown, the application package can include applications such as intelligent voice interaction, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, games, shopping, travel, and instant messaging (such as SMS). Additionally, the application package may also include system applications such as the home screen (desktop), the negative one screen, the control center, and the notification center.

[0183] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0184] like Figure 14 As shown, the application framework layer may include an input manager, window manager, content provider, view system, phone manager, resource manager, notification manager, display manager, activity manager, etc. For ease of explanation, Figure 14 In this example, the application framework layer includes an input manager, window manager, content provider, view system, and activity manager. It should be noted that any two modules within the input manager, window manager, content provider, view system, and activity manager can call each other.

[0185] The input manager is used to receive instructions or requests reported by lower layers such as the kernel layer and hardware abstraction layer.

[0186] A window manager is used to manage window programs. The window manager can obtain the screen size, determine if a status bar exists, lock the screen, and capture the screen, etc. In this application, the window manager is used to display a window including one or more shortcut controls when the electronic device 100 meets preset triggering conditions.

[0187] The Activity Manager is used to manage the running activities in the system, including information on processes, applications, services, and tasks.

[0188] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0189] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images. In this application, the view system is used to display a shortcut area on the display screen 1394 when the electronic device 100 meets preset trigger conditions. This shortcut area includes one or more shortcut controls added to the electronic device 100. This application does not limit the location or layout of the shortcut area, or the icons, locations, layouts, and functions of the controls within the shortcut area.

[0190] The display manager is used to transmit display content to the kernel layer.

[0191] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection, hang-up, etc.).

[0192] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0193] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating electronic devices, and flashing indicator lights.

[0194] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0195] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0196] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0197] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0198] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0199] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0200] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0201] A 2D graphics engine is a graphics engine for 2D drawing.

[0202] The Hardware Abstraction Layer (HAL) serves as the interface between operating system software and hardware components, providing a platform for interaction between upper-layer software and lower-layer hardware. The HAL abstracts the underlying hardware into software containing corresponding hardware interfaces. By accessing the HAL, settings can be configured for the underlying hardware devices; for example, enabling or disabling relevant hardware components can be done within the HAL. In some embodiments, the core architecture of the HAL layer is constructed using at least one of C++ or C++.

[0203] The kernel layer is the layer between hardware and software. The kernel layer includes at least the display driver, camera driver, audio driver, sensor driver, touch chip driver, and input system. For ease of explanation, Figure 14 In this example, the kernel layer includes the input system, touch chip driver, display driver, and storage driver. The display driver and storage driver can be jointly configured within the driver module.

[0204] It is understood that the structure illustrated in this application does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0205] This application provides a computer program product that, when run on a device, causes the device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0206] This application provides a readable storage medium containing instructions that, when executed on a device, cause the device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0207] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0208] It is understood that the various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0209] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0210] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. The mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0211] The above describes the possible hardware structures of electronic devices. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0212] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0213] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0214] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An image recognition method, characterized in that, Applied to electronic devices, including: Obtain the first image to be processed; The first image is subjected to multi-scale downsampling processing to obtain multiple downsampled images; Edge detection and difference comparison processing are performed on each of the multiple downsampled images to obtain a local information map corresponding to each downsampled image; Based on the local information map corresponding to each downsampled image, multiple target image blocks are determined from the multiple downsampled images, and the target image blocks include the valid information in the first image; Based on the plurality of target image blocks, image information in the first image is identified; The step of performing edge detection and difference comparison processing on each of the multiple downsampled images to obtain a local information map corresponding to each downsampled image includes: Edge detection processing is performed on each downsampled image to obtain the edge information corresponding to each downsampled image, and difference comparison processing is performed on the pixels in each downsampled image to obtain the neighborhood information corresponding to each downsampled image; The edge information and neighborhood information corresponding to each downsampled image are weighted and fused to obtain the local information map corresponding to each downsampled image; The step of determining multiple target image patches from the multiple downsampled images based on the local information map corresponding to each downsampled image includes: The local information map corresponding to each downsampled image is binarized to obtain the binary map corresponding to each downsampled image; The binary image corresponding to each downsampled image is divided into multiple sampled image blocks based on a preset size; The image blocks in the corresponding downsampled image that are sampled image blocks whose number of pixel values ​​is greater than the first value threshold are identified as target image blocks.

2. The method according to claim 1, characterized in that, The neighborhood information corresponding to each downsampled image is determined in the following way: Determine the variance of pixel values ​​for each pixel and associated pixels in each downsampled image; Based on the variance of pixel values ​​of each pixel and associated pixels, the neighborhood information corresponding to each downsampled image is determined.

3. The method according to claim 1, characterized in that, The step of binarizing the local information map corresponding to each downsampled image to obtain the binary map corresponding to each downsampled image includes: The local information map corresponding to each downsampled image is binarized according to a preset pixel threshold to obtain a binary map corresponding to each downsampled image.

4. The method according to claim 3, characterized in that, The step of binarizing the local information map corresponding to each downsampled image according to a preset pixel threshold to obtain a binary map corresponding to each downsampled image includes: In each downsampled image, the pixel values ​​greater than the pixel threshold are determined as a first value, and the pixel values ​​less than or equal to the pixel threshold in each downsampled image are determined as a second value, thereby obtaining a binary image corresponding to each downsampled image.

5. The method according to any one of claims 1 to 4, characterized in that, The multiple downsampled images include a second image, the size of which is smaller than the size of the first image. The step of determining the image block in the corresponding downsampled image where the number of sampled image blocks with pixel values ​​greater than a first value is greater than a threshold as the target image block includes: If the first sampled image block in the binary image corresponding to the second image satisfies the first preset condition, the first image block corresponding to the first sampled image block in the second image is determined as the target image block; The first preset condition is that the number of pixels with a pixel value of a first value is greater than a number threshold.

6. The method according to any one of claims 1 to 4, characterized in that, The multiple downsampled images include a second image and a third image, wherein the size of the second image is smaller than the size of the first image, and the size of the third image is smaller than the size of the second image. The step of determining the image block in the corresponding downsampled image where the number of sampled image blocks with pixel values ​​greater than a first value is greater than a threshold as the target image block includes: If the second sampled image block in the binary image corresponding to the second image does not meet the first preset condition, and the third sampled image corresponding to the second sampled image in the binary image corresponding to the third image meets the first preset condition, then the downsampled image block corresponding to the third sampled image block in the third image is determined as the target image block. The first preset condition is that the number of pixels with a pixel value of a first value is greater than a number threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The step of identifying image information in the first image based on the plurality of target image blocks includes: Based on the position information, size information, and image feature information of each target image block in the plurality of target image blocks, the image information in the first image is identified; The location information is determined based on the position of each target image block in the first coordinate system corresponding to the first image; the size information is determined based on the size of the downsampled image to which each target image block belongs; and the image feature information is obtained by extracting features from each target image block.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Receive the user's first instruction for the first image; Based on the image information of the first image, execute the first operation corresponding to the first instruction.

9. An electronic device, characterized in that, Includes: a memory for storing instructions executed by one or more processors of the electronic device. And a processor, which is one or more processors of the electronic device, for performing the image recognition method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the image recognition method according to any one of claims 1 to 8.

11. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by an electronic device, enable the electronic device to perform the image recognition method as described in any one of claims 1 to 8.

12. A chip, characterized in that, The chip includes a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the image recognition method according to any one of claims 1 to 8.

Images