Image processing method and device

Abstract

The invention discloses an image processing method and device, and the method comprises the steps: obtaining a first image; the first image is obtained by intercepting a video frame output when the electronic equipment runs a target application; processing the first image based on target information to obtain a second image; the target information is related to application information of the target application, and the second image is a partial region image of the first image; inputting the second image to a target model; the reasoning result of the target model for the first image is consistent with the reasoning result of the target model for the second image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and relate to but are not limited to an image processing method and device. Background Art

[0002] Artificial Intelligence (AI) is developing rapidly, and visual recognition based on neural networks is widely used. The object recognition process is simply a large number of convolution operations to extract features and then identify objects. This process requires a lot of computing power, that is, the computing power of the central processing unit (CPU) and the graphic processing unit (GPU). In the game scene, the game also requires a lot of CPU & GPU resources, so when using AI visual model recognition in the game scene, the efficiency of the AI ​​model has a great impact on the frame rate and smoothness of the game. How to improve the recognition efficiency of the AI ​​model has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, an embodiment of the present application provides an image processing method and device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Obtaining a first image; the first image is obtained by intercepting a video frame output when the electronic device runs a target application;

[0007] Processing the first image based on target information to obtain a second image; the target information is related to application information of the target application, and the second image is an image of a partial area of ​​the first image;

[0008] The second image is input to the target model; the inference result of the target model for the first image is consistent with the inference result for the second image.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0010] A first acquisition module is used to obtain a first image; the first image is obtained by intercepting a video frame output when the electronic device runs a target application;

[0011] A first processing module is used to process the first image based on target information to obtain a second image; the target information is related to application information of a target application, and the second image is an image of a partial area of ​​the first image;

[0012] The input module is used to input the second image into the target model; the reasoning result of the target model for the first image is consistent with the reasoning result for the second image.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, the memory storing a computer program that can be run on the processor, the processor executing to obtain a first image; the first image is obtained by intercepting a video frame output when the electronic device runs a target application; the first image is processed based on target information to obtain a second image; the target information is related to application information of the target application, and the second image is an image of a partial area of ​​the first image; the second image is input into a target model; the inference result of the target model for the first image is consistent with the inference result for the second image.

[0014] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for implementing the above method when executed by a processor.

[0015] In a fifth aspect, a computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in the above method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A A schematic diagram of a process for AI reasoning and identifying game information provided in an embodiment of the present application;

[0017] Figure 1B A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0018] Figure 2A A schematic diagram of an implementation flow of processing a first image provided in an embodiment of the present application;

[0019] Figure 2B A schematic diagram of an implementation flow of processing a first image provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of an implementation process of identifying game highlight moments provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of clipping a partial image provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0023] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0025] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0028] Figure 1A A flowchart of an AI reasoning and identifying game information provided in an embodiment of the present application can be implemented by the following steps:

[0029] Step S1, capturing screen content;

[0030] Here, the resolutions of different electronic devices are different. Take an electronic device with a 2K resolution as an example, (2K = 2560 * 1600), obtain an image of 2560 * 1600, and then convert the image format. For example, the image format that can be captured is NV12, and NV12 is converted to RGB format. Among them, NV12 is a YUV format used for the storage and transmission of video data. YUV is a color encoding method, in which Y represents luminance or brightness, and U and V are chrominance or chromaticity, which describes the color and saturation. The NV12 format is suitable for video processing because it can efficiently store and transmit the brightness and chromaticity information of video frames. RGB is an additive color model used to represent colors. Various colors are generated based on different intensities of the three basic colors of red, green, and blue. In the RGB format, the color of each pixel is determined by the values ​​of the three components R, G, and B.

[0031] Step S2, unifying the resolution;

[0032] Here, taking the target resolution of 1K (1K=1920*1080) as an example, an image of 2560*1600 is compressed to 1920*1080.

[0033] Step S3, reducing the image;

[0034] During the implementation process, the 1920*1080 image can be reduced to a 640*640 image; and then the 640*640 image can be recognized or classified.

[0035] In the embodiment of the present application, all operations (format conversion, scaling, etc.) are performed on all data in the entire original image during the entire process, which increases the operating burden of the processor. How to effectively reduce the operating burden of the processor when recognizing the original image has become a technical problem that needs to be solved urgently.

[0036] The present application embodiment provides an image processing method, such as Figure 1B As shown, the method includes:

[0037] Step S110, obtaining a first image; the first image is obtained by intercepting a video frame output when the electronic device runs a target application;

[0038] Here, the target application may be a game application, a live broadcast application, or a video software application. The first image is a video frame captured when the target application is running.

[0039] During the implementation process, the running screen of the target application during operation can be captured by using screenshot software through the screen or program window, or the running screen of the target application during operation can be obtained by reading video content.

[0040] Step S120: Process the first image based on target information to obtain a second image; the target information is related to application information of the target application, and the second image is an image of a partial area of ​​the first image;

[0041] Here, the target information is information related to the application information of the target application, including but not limited to the application type, application identifier, application name and other application information of the target application. For example, the target information can be the name of the game. Since the position of each game feature is basically fixed in one area, such as the kill information of the first game is in the upper right corner area, the kill information of the second game is in the lower center of the screen, and the wonderful information of the third game is in the upper center of the screen. During implementation, the position information of the second image in the first image can be determined based on the name of the game, and then the second image can be captured from the first image based on the position information, and the second image can be format converted and scaled so that the second image can meet the size and format requirements of the input target model for processing.

[0042] Step S130, inputting the second image into the target model; the inference result of the target model for the first image is consistent with the inference result for the second image.

[0043] Here, the target model may be an AI model that can intelligently recognize images. During implementation, the target model can be used to infer the recognition result of the second image, and the obtained inference result is consistent with the inference result of the first image. For example, an AI model can be used to identify a local area in a game interface, such as a skill control area, to infer whether features such as kills appear in the game interface. During implementation, since the inference result obtained using the local area is the same as the recognition result obtained by recognizing the entire game interface, recognizing the local area can effectively reduce the computing power burden of the processor without affecting the recognition result, thereby improving the recognition efficiency of the processor.

[0044] In the embodiment of the present application, the first image is first obtained by intercepting the video frame output when the electronic device runs the target application; then the first image is processed based on the target information related to the application information of the target application to obtain a partial area image of the first image, that is, the second image; finally, the second image is input into the target model to use the target model to perform recognition and reasoning on the second image to obtain the reasoning result. In this way, since the second image is a partial area image of the first image, the target model is used to perform recognition and reasoning only on the second image. Compared with performing recognition and reasoning on the first image, without affecting the reasoning result, the computing power used by the target model in recognizing and reasoning the image can be greatly saved, and the efficiency of the target model in image recognition and reasoning can be effectively improved.

[0045] In some embodiments, in the above step S120, “processing the first image based on the target information to obtain the second image”, Figure 2A As shown, this can be achieved by following the steps below:

[0046] Step S201: if the target information indicates that the target application corresponds to at least one type of target feature and the feature positions of all target features are the same, intercepting a second image from the first image based on the feature position;

[0047] Here, a target application may have at least one type of target feature, and all of the above target features may be located at the same feature position of the first image. For example, a game screen may have two different types of features, namely, kill and escape, and the feature position area corresponding to the above two different features may be, for example, a fixed area in the lower right corner, and the second screen located in the same feature position area (fixed area in the lower right corner) of the game screen may be used to identify whether the above two features appear.

[0048] The above target features can enable the target model to infer the second image based on the recognition of the target features after the second image is input into the target model.

[0049] Step S202: if the target information indicates that the target application corresponds to at least two types of target features and the feature positions of the at least two types of target features are different, a plurality of sub-images are intercepted from the first image based on each of the feature positions, and the plurality of sub-images are merged to obtain the second image;

[0050] Here, a target application may have at least two types of target features, and the feature positions of at least two types of target features are different.

[0051] During implementation, multiple sub-images corresponding to multiple features can be captured from the first image based on each of the feature positions, and the multiple sub-images can be merged to obtain a second image. For example, two different types of features, namely, kill and escape, can exist in a game screen, and the identifiable feature position areas corresponding to the above two different features are different, which can be a fixed area in the lower right corner and a fixed area in the upper left corner. Two sub-images of different feature position areas (the fixed area in the lower right corner and the fixed area in the upper left corner) located in the game screen can be obtained, and the two sub-images are merged to obtain the second image.

[0052] So that the target model can infer the second image based on the target features.

[0053] In an embodiment of the present application, if the target information represents that the target application corresponds to at least one type of target feature and the feature positions of all target features are the same, a second image is captured from the first image based on the feature positions; in this way, it is possible to obtain a second image based on a feature position to identify the second image and determine whether at least one type of target feature appears in the first image.

[0054] If the target information represents that the target application corresponds to at least two types of target features and the feature positions of the at least two types of target features are different, multiple sub-images are captured from the first image based on each feature position, and the multiple sub-images are merged to obtain the second image; in this way, sub-images corresponding to features at different positions in the first image can be obtained, and a second image capable of identifying features at different positions can be obtained, so as to identify the second image and determine whether at least two types of target features corresponding to different positions appear in the first image.

[0055] In some embodiments, in the above step S120, “processing the first image based on the target information to obtain the second image”, for example Figure 2B As shown, this can be achieved by following the steps below:

[0056] Step S210: if the target information indicates that the target object has performed a target operation on the target application, determining a feature position of at least one type of target feature selected by the target operation;

[0057] Here, the target operation may be a selection operation made by the user in response to the feature of the target application. For example, a game has multiple different features such as killing, continuous shooting, and escaping, and the target operation may be the user's selection to process the killing feature in the game.

[0058] In the implementation process, after the target operation is performed, the feature position of at least one type of selected target feature can be determined, that is, some features of the multiple features corresponding to the target application can be selected as the target features to be processed.

[0059] Step S220: when the target operation selects a type of target feature, intercepting the second image from the first image based on the feature position of the selected target feature;

[0060] In the implementation process, the feature position of the selected target feature in the first image can be determined first, and then the second image can be intercepted from the first image based on the feature position. For example, a kill in a certain game can be selected as the target feature, and the location where the kill occurs can be located in the upper right corner of the first image, and then the upper right corner of the first image can be intercepted as the second image.

[0061] Step S230: when at least two types of target features are selected in the target operation, a plurality of sub-images are intercepted from the first image based on the feature positions of each type of target features selected, and the plurality of sub-images are merged to obtain the second image;

[0062] In some embodiments, feature positions corresponding to at least two types of target features may be located in different regions of the first image, and multiple sub-images may be captured from the corresponding different regions of the first image, and the multiple sub-images may be spliced ​​to obtain the second image.

[0063] In some embodiments, feature positions corresponding to at least two types of target features may be located in the same area of ​​the first image, and the second image may be captured from the corresponding target area in the first image.

[0064] So that the target model can infer the second image based on the target features.

[0065] In an embodiment of the present application, if the target information represents that the target object has performed a target operation on the target application, the feature position of at least one type of target feature selected by the target operation is determined; if the target operation selects one type of target feature, the second image is intercepted from the first image based on the feature position of the selected target feature; if the target operation selects at least two types of target features, multiple sub-images are intercepted from the first image based on the feature position of each type of target feature selected, and the multiple sub-images are merged to obtain the second image; so that the target model can infer the second image based on the target feature. In this way, a selection operation can be performed on multiple target features of the target application to obtain the feature area corresponding to the selected feature to intercept the second image, thereby realizing the recognition and inference of the selected feature.

[0066] In some embodiments, the target information also includes a first size of the input image required by the target model; the "cutting out a plurality of sub-images from the first image" in the above step S202 or step S230 can be implemented by the following process:

[0067] The second size of each sub-image is determined based on the first size, and the corresponding sub-image is intercepted from the first image according to the second size, so that the second image obtained by merging the multiple sub-images has the first size.

[0068] Here, the target model's requirements for the input image size mainly depend on the model, algorithm, and specific application scenario used. Most image recognition models, especially deep learning models such as Convolutional Neural Networks (CNN), usually require fixed-size images for their input layer. This is because the weights and architecture of the model are determined before training, and the number of weight parameters is directly related to the dimensions of the input layer. For example, AlexNet requires the input image size to be 227×227 pixels, VGG requires 224×224 pixels, InceptionV3 requires 299×299 pixels, and ResNet also requires 224×224 pixels. Among them, AlexNet is a classic convolutional neural network; Visual Geometry Group (VGG) is a deep convolutional neural network architecture; InceptionV3 is a deep learning model for image recognition and classification, belonging to the third version of the Inception series of models; Residual Neural Network is a deep neural network structure.

[0069] Assuming that there are two types of target features, the number of sub-images to be captured is 2, and the target model is ResNet (requires input of 224×224 pixel images). At this time, the capture size of each sub-image can be set to 112×112 pixels (here it is assumed that the size of each type of target feature is less than 112×112 pixels), and the size of the second image obtained by merging the two sub-images is 112×112 pixels, so that it can be directly input into the model for processing without other scaling, cropping and other steps (in this case, the model only needs to infer one image, but can take into account both types of target features at the same time). Compared with capturing two images, scaling them to 112×112 pixels and then inputting them into the model (in this case, the model needs to infer two images. If one of the images does not contain the target features, then inference on this image is useless), it is more economical in terms of device computing power.

[0070] In the embodiment of the present application, the target information also includes the first size of the input image required by the target model; the second size of each sub-image is determined based on the first size, and the corresponding sub-image is intercepted from the first image according to the second size, so that the second image obtained by merging multiple sub-images has the first size. In this way, the second image with the first size is obtained, which can effectively improve the efficiency and accuracy of the target model in inferring the second image.

[0071] In some embodiments, the target information further includes a first size of the input image required by the target model and a third size of the image corresponding to each type of target feature; the "capturing multiple sub-images from the first image" in the above step S202 or step S230 can be implemented by the following steps:

[0072] Step A, determining a corresponding fourth size based on the first size and each third size; the fourth size is not less than the third size;

[0073] Here, the target information includes the first size required by the target model and the third size corresponding to each type of target feature. Different target features may correspond to the same third size or different third sizes. For example, in a game, the size of the kill feature may be the same as the size of the escape feature; or the size of the kill feature may be different from the size of the escape feature.

[0074] During the implementation process, the fourth size corresponding to each third size can be determined based on the first size and each third size, and the fourth size is not less than the third size, so that the acquired sub-image includes the target features. Under the constraint of the first size, it is also ensured that the size of the merged second image is the same as the first size or the difference is not much.

[0075] Step B: intercepting corresponding sub-images from the first image according to the fourth size, so that a second image obtained by merging the multiple sub-images has the first size.

[0076] During implementation, sub-images may be respectively captured from the first image based on the fourth size, and the plurality of sub-images may be combined to obtain a second image having the first size.

[0077] In some embodiments, for each type of target feature, if the area occupied by the feature is large, the corresponding fourth size is large, and if the area occupied by the feature is small, the fourth size of the object is small. At the same time, the fourth size also depends on the first size. If the first size is large, the corresponding fourth size is large, and if the first size is small, the corresponding fourth size is large. It is necessary to take both the first size and the third size into consideration at the same time, so that the second image has the first size or is not much different from the first size, and the sub-image corresponding to each feature can occupy an appropriate proportion in the second image, thereby improving the accuracy of the target model in recognizing each feature in the second image.

[0078] Assuming that there are four types of target features, the number of sub-images to be captured is 4, and the target model is ResNet (requiring an input image of 224×224 pixels). At this time, the capture size of each sub-image can be set to 56×56 pixels, but the first and fourth types of target features are smaller (both are 10×10), and the second and third types of target features are larger (90×90). If the sub-images of the first and fourth types of target features are captured at 56×56, the proportion of the features in the sub-image will be smaller, resulting in the feature not being recognized or being recognized incorrectly. Similarly, if the sub-images of the second and third types of target features are captured at 56×56, the sub-image will not include all features, resulting in the feature not being recognized or being recognized incorrectly. Therefore, the cut-off size can be appropriately adjusted with reference to the size of the feature. After balancing, the cut-off size (that is, the fourth size) of the first and second target features can be set to 20×20, and the cut-off size (that is, the fourth size) of the second and third target features can be set to 102×102.

[0079] It can be understood that if the size of the merged second image cannot be the same as the first size, it can also be merged into an image with a size that is not much different from the first size. On this basis, it is only necessary to scale the second image by a small proportion to make the size of the merged second image the same as the first size.

[0080] In the embodiment of the present application, the target information also includes the first size of the input image required by the target model and the third size of the image corresponding to each type of target feature; firstly, the corresponding fourth size is determined based on the first size and each third size; the fourth size is not less than the third size; then, the corresponding sub-image is intercepted from the first image according to the fourth size, so that the second image obtained by merging multiple sub-images has the first size. In this way, when identifying multiple types of target features, it can be achieved that the sub-image corresponding to each feature included in the second image can occupy a suitable size in the second image, thereby improving the accuracy of the target model in identifying each feature in the second image.

[0081] In some embodiments, the above image processing method further includes the following steps:

[0082] Step S140: obtaining a preset pixel value normalization mapping table, wherein the pixel value normalization mapping table can provide a mapping table relationship between original pixel values ​​and normalized pixel values;

[0083] Here, the pixel values ​​of the image data can correspond to the range of 0 to 255. Before reasoning, all pixel data needs to be converted to between 0 and 1. This process is called normalization. Normalization can make the input data have a uniform scale in terms of values, avoid the problem of large numerical differences between different features, and speed up the convergence of the training network.

[0084] Currently, when RGB data is normalized to a floating point number between 0 and 1, the RGB value is directly divided by 255.0f, which is a very time-consuming process. For example, before the improvement, when each pixel data needs to be normalized to a floating point number between 0 and 1, the RGB value is directly divided by 255.0f. The normalization operation for a 1920*1080 image takes 44 milliseconds (ms).

[0085] During implementation, a pixel value normalization mapping table that can provide a mapping table relationship between original pixel values ​​and normalized pixel values ​​can be pre-set. The pixel normalization mapping table includes a normalization value corresponding to each pixel value in the range of 0 to 255 pixel values, for example, 0 corresponds to 0, 1 corresponds to 0.0039, 2 corresponds to 0.0078, 3 corresponds to 0.0117, ..., 255 corresponds to 1.

[0086] Step S150: performing normalization processing on each pixel point in the second image based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each pixel point;

[0087] During implementation, normalization processing is performed on each pixel point in the second image based on the pixel value normalization mapping table, and a normalized pixel value in the range of 0 to 1 corresponding to RGB can be directly obtained without repeated calculation.

[0088] For example, after the improvement, the normalization operation time for a 1920*1080 image is 19 milliseconds (ms), which is 44 milliseconds before the improvement. The efficiency of the operation has obviously been improved.

[0089] Step S160: Determine the inference result by using the target model based on the normalized pixel value corresponding to each pixel point in the second image.

[0090] During implementation, all pixel data in the second image may be converted to values ​​between 0 and 1 using a pixel value normalization mapping table before reasoning, thereby achieving reasoning of the second image using the target model.

[0091] In the embodiment of the present application, firstly, a preset pixel value normalization mapping table is obtained, wherein the pixel value normalization mapping table can provide a mapping table relationship between the original pixel value and the normalized pixel value; then, each pixel point in the second image is normalized based on the pixel value normalization mapping table to obtain the normalized pixel value corresponding to each pixel point; finally, the target model is used to determine the inference result based on the normalized pixel value corresponding to each pixel point in the second image. In this way, after the sampling data is mapped to the lookup table, the division operation is reduced, the amount of operation of the second image is reduced during the normalization operation, and the time consumption is greatly reduced.

[0092] In some embodiments, the above step S150 "normalizing each pixel in each of the target sub-images based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each pixel" can be implemented by the following process:

[0093] The superscalar of the central processing unit is used to perform parallel normalization processing on each pixel point in each of the target sub-images based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each of the pixel points.

[0094] Here, the superscalar CPU refers to a type of parallel computing that implements instruction-level parallelism in a processor core. This technology can achieve higher CPU throughput at the same CPU main frequency. The core of the processor generally has multiple execution units (or functional units), such as arithmetic logic units (ALUs), shift units, multipliers, etc. The CPU with a superscalar architecture can dispatch multiple instructions to be executed in different execution units at the same time in one clock cycle, achieving instruction-level parallelism. Since the CPU is a pipeline mechanism, the sampling loop is expanded, the specified parallelism is increased, and the dependency between parallel instructions is reduced, so that the CPU superscalar technology can be fully utilized and the execution speed can be improved.

[0095] In the embodiment of the present application, the superscalar of the central processing unit is used to perform parallel normalization processing on each pixel point in each target sub-image based on the pixel value normalization mapping table to obtain the normalized pixel value corresponding to each pixel point. In this way, the superscalar of the central processing unit is used, and the multi-stage pipeline method without mutual dependence is used to increase the parallel operation process, effectively improving the operation efficiency.

[0096] In some embodiments, the above step S120 "processing the first image based on the target information to obtain the second image" can be implemented by the following steps:

[0097] Step 121, obtaining second position information of the second image in the third image that meets a preset resolution;

[0098] Here, the preset resolution may be the most common resolution, such as 1080P. It may also be a preset resolution set by a user based on the screen resolution of a commonly used electronic device. In the implementation process, the second position information of the second image corresponding to the target feature to be identified in the third image that meets the preset resolution may be determined. For example, the position information of the killing feature in the 1080P image may be obtained.

[0099] Step 122: determining first position information of the second image in the first image based on a ratio of a resolution of the first image to the preset resolution and the second position information;

[0100] Here, the resolution of the first image may be different from the preset resolution or the same as the preset resolution. When the first image has the same resolution as the preset resolution, the ratio is 1, and the first position information of the second image in the first image is the same as the second position information of the second image in the third image. When the first image has a different resolution from the preset resolution, the ratio of the resolution of the first image to the preset resolution may be obtained, and then the first position information of the second image in the first image may be determined based on the ratio and the second position information.

[0101] For example, the position information of the game feature at the preset resolution of 1K (1920*1080) is Rect={x0, y0, 640, 640}, and the original captured image is an electronic device with a resolution of 2K (2560*1600), and an image of size 2560*1600 is obtained. The position information of this position in the original captured image is calculated, that is, the position information Rect1={x1, y1, 640*r1, 640*r2} in the image obtained based on the screen with different resolutions, where r1=2560 / 1920 and r2=1600 / 1080.

[0102] Step 123: capture the second image in the first image based on the first position information.

[0103] In the implementation process, after determining that the target feature in the first image corresponds to the first position information, a screenshot of the second image corresponding to the first position information is taken. Next, the Rect1 area of ​​the original image can be directly cropped and the data format converted and scaled. The entire process only preprocesses the final feature area data, which can greatly improve the speed of data reasoning.

[0104] In the embodiment of the present application, the second position information of the second image in the third image that meets the preset resolution is first obtained; then, based on the ratio of the resolution of the first image to the preset resolution and the second position information, the first position information of the second image in the first image is determined; finally, based on the first position information, the second image is intercepted in the first image. In this way, the second position information corresponding to the target feature in the third image of the preset resolution can be used to determine the first position information of the target feature corresponding to the target feature in the first image of different resolutions, so as to intercept the second image corresponding to the target feature in the first image.

[0105] In some embodiments, the above image processing method further includes the following steps:

[0106] Step S170: Determine a second image set that meets the target condition based on the output of the target model, and generate a target video based on the second image set.

[0107] Here, the target condition may be that the second image has a target feature, and each frame of the generated target video includes an image having the target feature. For example, when the target application is a game program, a set of images having a kill feature may be obtained to generate a video collection having a kill highlight moment.

[0108] In the embodiment of the present application, a set of second images satisfying the target condition is determined based on the output of the target model, and a target video is generated based on the set of second images. In this way, when it is determined based on the target model that the second image satisfies the target condition, a target video of all second images satisfying the target condition can be generated, so that the second images satisfying the target condition can be continuously viewed.

[0109] Figure 3 A schematic diagram of a process flow for identifying game highlight moments provided in an embodiment of the present application can be implemented by the following steps:

[0110] Step S310, obtaining a game content image;

[0111] During the implementation process, the running screen of the game can be captured through the screen or program window, or the running screen of the game can be obtained by reading the game video content.

[0112] Step S320: pre-process only the target area in combination with the characteristic rules;

[0113] Here, the position of each game feature is basically fixed in one area. For example, the kill information of the first game is in the upper right corner, the kill information of the second game is in the lower center of the screen, and the wonderful information of the third game is in the upper center of the screen.

[0114] During the implementation process, different games need to identify different feature positions, and each game can set a corresponding target position information to be cropped; based on the target position information, combined with the currently acquired entire screen content information, the entire screen is cropped to obtain the feature screen corresponding to the target position. After that, the feature screen is format converted and scaled, that is, the entire process only operates the content of the target area, and does not operate the final non-target cropping area, which can reduce a lot of calculations and improve the computing efficiency of the CPU & GPU.

[0115] Figure 4 A schematic diagram of cutting a partial image provided in an embodiment of the present application is shown in FIG. Figure 4 As shown in FIG. 1 , the schematic diagram includes an image whose format is NV12 and whose resolution is 2560*1600, and an image whose format is RGB and whose resolution is 640*640 after cropping. In the implementation process, only the cropped image is processed subsequently, which can reduce a lot of calculations.

[0116] During the implementation process, the position information of the final cropped area of ​​different game features, that is, the position information of the game features in the preset resolution image Rect = {x0, y0, 640, 640}, can be calculated to obtain the position information of this position in the original captured image, that is, the position information Rect1 = {x1, y1, 640*r1, 640*r2} in the image acquired based on different resolution screens, where r1 = 2560 / 1920 and r2 = 1600 / 1080. Next, the Rect1 area of ​​the original image can be directly cropped and the data format converted and scaled. The entire process only preprocesses the final feature area data, which can greatly improve the speed of data AI reasoning.

[0117] In some embodiments, when using AI to identify exciting moments in games, it is found that many game features are very small in size. If an original image is directly used for inference, too much feature information will be lost after multiple downsampling convolutions of the neural network, resulting in a decrease in recognition accuracy. For example, a 640*640 image is convolved multiple times, and the resulting 80*80, 40*40, and 20*20 images are used for inference. If the size of the feature is very small, too much feature information will be lost after multiple downsampling convolutions of the neural network.

[0118] During the implementation process, a reasonable scaling ratio may be established based on the size of the feature, and an input image with sufficient feature information may be provided to the AI ​​model to obtain a high recognition rate. For example, during the inference process, the AI ​​model can generate feature extraction results of three sizes (large, medium, and small) for classification (x*x, y*y, and z*z). Based on the actual feature size in different games and the amount of feature information after feature extraction of different sizes during the inference process, a scaling relationship is established, thereby improving the feature information during the inference process and improving the accuracy.

[0119] Here, the target size with high inference accuracy is larger than m*n. It can be cropped and resized to a size with high accuracy according to the characteristic target position and size of different games, which can improve the recognition accuracy.

[0120] Step S330, optimizing the normalization operation;

[0121] Here, the pixel values ​​of the image data can correspond to the range of 0 to 255. Before reasoning, all pixel data needs to be converted to between 0 and 1. This process is called normalization. Normalization can make the input data have a uniform scale in terms of values, avoid the problem of large numerical differences between different features, and speed up the convergence of the training network. Currently, when normalizing RGB data to a floating point number between 0 and 1, the RGB value is directly divided by 255.0f, which is a very time-consuming process.

[0122] After the sampled data is mapped to the lookup table, the division operation is reduced, and the multi-stage pipeline mode without mutual dependence is used. The time consumed in the normalization operation of a 1080P image is greatly reduced. The core of the optimized normalization process is to reduce the amount of calculation and increase the parallel operation process.

[0123] Step S340: AI model object recognition.

[0124] During the implementation process, due to the feature distribution rules of different recognition targets, feature images at different positions are cropped, which reduces feature loss during image scaling and improves the accuracy of target recognition.

[0125] In the embodiment of the present application, according to the RGB data distribution, an expansion mechanism is adopted, and the instruction pipeline advantage and lookup table mechanism of the CPU are utilized to sample and improve the performance of the RGB image normalization operation by 100%. According to the regional information of the current game feature, only the data in the region is converted, and various operations of the data outside the region are removed, which greatly improves the efficiency. The sizes of different recognition targets are adjusted to improve the recognition accuracy.

[0126] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes various modules, each module includes various sub-modules, each sub-module includes a unit, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0127] Figure 5 A schematic diagram of the structure of the image processing device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 includes:

[0128] A first obtaining module 510 is used to obtain a first image; the first image is obtained by intercepting a video frame output when the electronic device runs a target application;

[0129] A first processing module 520 is configured to process the first image based on target information to obtain a second image; the target information is related to application information of the target application, and the second image is an image of a partial area of ​​the first image;

[0130] The input module 530 is used to input the second image into the target model; the inference result of the target model for the first image is consistent with the inference result for the second image.

[0131] In some embodiments, the first processing module 520 includes a first interception submodule and a second interception submodule, wherein the first interception submodule is used to intercept a second image from the first image based on the feature position if the target information represents that the target application corresponds to at least one type of target feature and the feature positions of all target features are the same; the second interception submodule is used to intercept multiple sub-images from the first image based on each feature position if the target information represents that the target application corresponds to at least two types of target features and the feature positions of at least two types of target features are different, and merge the multiple sub-images to obtain the second image; so that the target model can infer the second image based on the target feature.

[0132] In some embodiments, the first processing module 520 includes a first determination submodule, a first interception submodule and a second interception submodule, wherein the first determination submodule is used to determine the feature position of at least one type of target feature selected by the target operation if the target information represents that the target object has performed a target operation on the target application; the first interception submodule is used to intercept the second image from the first image based on the feature position of the selected target feature when the target operation selects one type of target feature; the second interception submodule is used to intercept multiple sub-images from the first image based on the feature position of each type of target feature selected when the target operation selects at least two types of target features, and merge the multiple sub-images to obtain the second image; so that the target model can infer the second image based on the target feature.

[0133] In some embodiments, the target information also includes a first size of the input image required by the target model; the second cropping submodule is further used to determine a second size of each sub-image based on the first size, and crop a corresponding sub-image from the first image according to the second size, so that the second image obtained by merging the multiple sub-images has the first size.

[0134] In some embodiments, the target information also includes a first size of the input image required by the target model and a third size of the image corresponding to each type of target feature; the second interception submodule includes a determination unit and a interception unit, wherein the determination unit is used to determine a corresponding fourth size based on the first size and each third size; the fourth size is not less than the third size; the interception unit is used to intercept a corresponding sub-image from the first image according to the fourth size, so that the second image obtained by merging the multiple sub-images has the first size.

[0135] In some embodiments, the device also includes a second acquisition module, a second processing module and a first determination module, wherein the second acquisition module is used to obtain a preset pixel value normalization mapping table, wherein the pixel value normalization mapping table can provide a mapping table relationship between the original pixel value and the normalized pixel value; the second processing module is used to normalize each pixel point in the second image based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each pixel point; the first determination module is used to use the target model to determine the inference result based on the normalized pixel value corresponding to each pixel point in the second image.

[0136] In some embodiments, the second processing module is further used to utilize the superscalar of the central processing unit to perform parallel normalization processing on each pixel point in each of the target sub-images based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each pixel point.

[0137] In some embodiments, the processing module 520 includes an acquisition submodule, a second determination submodule and a third interception submodule, wherein the acquisition submodule is used to acquire the second position information of the second image in the third image that meets the preset resolution; the second determination submodule is used to determine the first position information of the second image in the first image based on the ratio of the resolution of the first image and the preset resolution, and the second position information; and the third interception submodule is used to intercept the second image in the first image based on the first position information.

[0138] In some embodiments, the device further includes a generation module for determining a second image set satisfying a target condition based on an output of the target model, and generating a target video based on the second image set.

[0139] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0140] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0141] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the image processing method provided in the above embodiment are implemented.

[0142] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and the processor 602 implements the steps of the image processing method provided in the above embodiment when executing the program.

[0143] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0144] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0145] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0148] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0150] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0151] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0152] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0153] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0154] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0155] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, the method comprising: obtaining a first image; The first image is obtained by intercepting a video frame output when the electronic device runs the target application; Processing the first image based on target information to obtain a second image; The target information is related to the application information of the target application, and the second image is a partial area image of the first image; inputting the second image into a target model; The inference result of the target model for the first image is consistent with the inference result for the second image.

2. The method according to claim 1, wherein the processing the first image based on the target information to obtain the second image comprises: If the target information indicates that the target application corresponds to at least one type of target feature and feature positions of all the target features are the same, intercepting a second image from the first image based on the feature position; If the target information indicates that the target application corresponds to at least two types of target features and the feature positions of the at least two types of target features are different, a plurality of sub-images are intercepted from the first image based on each of the feature positions, and the plurality of sub-images are combined to obtain the second image; So that the target model can infer the second image based on the target features.

3. The method according to claim 1, wherein the processing the first image based on the target information to obtain the second image comprises: If the target information represents that the target object has performed a target operation on the target application, determining a feature position of at least one type of target feature selected by the target operation; In a case where the target operation selects a type of target feature, intercepting the second image from the first image based on a feature position of the selected target feature; In a case where at least two types of target features are selected by the target operation, a plurality of sub-images are intercepted from the first image based on feature positions of each type of target features selected, and the plurality of sub-images are merged to obtain the second image; So that the target model can infer the second image based on the target features.

4. The method according to claim 2 or 3, wherein the target information further comprises a first size of the input image required by the target model; and the extracting a plurality of sub-images from the first image comprises: The second size of each sub-image is determined based on the first size, and the corresponding sub-image is intercepted from the first image according to the second size, so that the second image obtained by merging the multiple sub-images has the first size.

5. The method according to claim 2 or 3, wherein the target information further comprises a first size of the input image required by the target model and a third size of the image corresponding to each type of target feature; and extracting a plurality of sub-images from the first image comprises: determining a corresponding fourth size based on the first size and each third size; The fourth size is not less than the third size; Corresponding sub-images are intercepted from the first image according to the fourth size, so that a second image obtained by merging the multiple sub-images has the first size.

6. The method according to any one of claims 1 to 3, further comprising: Obtaining a preset pixel value normalization mapping table, wherein the pixel value normalization mapping table can provide a mapping table relationship between original pixel values ​​and normalized pixel values; Performing normalization processing on each pixel point in the second image based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each pixel point; The target model is used to determine the inference result based on the normalized pixel value corresponding to each pixel point in the second image.

7. The method according to claim 6, wherein the normalizing process is performed on each pixel point in each target sub-image based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each pixel point, comprising: The superscalar of the central processing unit is used to perform parallel normalization processing on each pixel point in each of the target sub-images based on the pixel value normalization mapping table to obtain a normalized pixel value corresponding to each of the pixel points.

8. The method according to claim 1, wherein processing the first image based on the target information to obtain the second image comprises: Acquire second position information of the second image in a third image that meets a preset resolution; Determining first position information of the second image in the first image based on a ratio of a resolution of the first image to the preset resolution and the second position information; The second image is captured in the first image based on the first position information.

9. The method according to any one of claims 1 to 3, further comprising: A second image set satisfying a target condition is determined based on an output of the target model, and a target video is generated based on the second image set.

10. An image processing device, comprising: An acquisition module, used for acquiring a first image; The first image is obtained by intercepting a video frame output when the electronic device runs the target application; A processing module, configured to process the first image based on target information to obtain a second image; The target information is related to the application information of the target application, and the second image is a partial area image of the first image; An input module, used for inputting the second image into a target model; The inference result of the target model for the first image is consistent with the inference result for the second image.

Images