Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of data compression and storage, and the image processing method comprises the steps: obtaining key feature data of a to-be-stored image in response to obtaining at least one frame of to-be-stored image; the to-be-stored image is coded to obtain a corresponding coded file, and the data size of the coded file is smaller than that of the to-be-stored image; and associating the encoding file with the key feature data, and storing the encoding file and the key feature data to a target storage area of the electronic equipment, so that the electronic equipment can obtain a target image from the target storage area in response to a target search demand.

Description

Technical Field

[0001] The present application belongs to the technical field of data compression and storage, and in particular, to an image processing method and device. Background Art

[0002] With the improvement of multimodal (text, image, voice, etc.) model capabilities, the use of multimodal technology to understand user intent has become a hot topic. Among them, for user personal terminals, image data storage is often required to support the identification and perception of user scenarios based on stored images. However, in actual applications, there is often a phenomenon of excessive image data storage, resulting in a high usage of storage resources and may affect the operating rate of terminal devices. Summary of the invention

[0003] To this end, this application discloses the following technical solutions:

[0004] An image processing method, comprising:

[0005] In response to obtaining at least one frame of image to be stored, acquiring key feature data of the image to be stored;

[0006] Performing encoding processing on the image to be stored to obtain a corresponding encoded file, wherein the data volume of the encoded file is smaller than the data volume of the image to be stored;

[0007] The encoding file is associated with the key feature data and stored in a target storage area of ​​the electronic device, so that the electronic device can obtain a target image from the target storage area in response to a target search requirement.

[0008] Optionally, obtaining key feature data of the image to be stored includes:

[0009] Identifying object content in the image to be stored, wherein the object content includes text and / or image;

[0010] Using the identified text recognition result and / or image recognition result as key feature data of the image to be stored; or,

[0011] The recognized text content and / or image content is vectorized, and the obtained vector data is used as key feature data of the image to be stored.

[0012] Optionally, obtaining key feature data of the image to be stored includes at least one of the following:

[0013] Identify the image content of the image to be stored, and use the obtained identification result or feature vector data as key feature data of the image to be stored;

[0014] Performing image-to-text processing on the image to be stored, and using target text data in the obtained text description data as key feature data of the image to be stored;

[0015] Obtaining image parameter data of the image to be stored, and using the image parameter data as key feature data of the image to be stored;

[0016] Obtaining operation behavior data on the image to be stored, and using the operation behavior data as key feature data of the image to be stored;

[0017] The source data of the image to be stored is obtained, and the source data is used as the key feature data of the image to be stored.

[0018] Optionally, obtaining at least one frame of image to be stored includes at least one of the following:

[0019] Monitoring the highlight moments of a target player in a target game application, and obtaining a screenshot image corresponding to the highlight moment as the image to be stored;

[0020] In response to a screenshot operation on a current application interface of the electronic device, taking a screenshot image obtained by the screenshot operation as the image to be stored;

[0021] In response to obtaining an image acquisition operation performed by a target user on an electronic device, using an image obtained by the image acquisition operation as the image to be stored;

[0022] In response to obtaining an image sharing request from a target device, using the received shared image as the image to be stored, wherein the target device is a device that establishes a target communication connection with the electronic device;

[0023] In response to obtaining the prompt word information input into the target raw image model, a generated image generated by the target raw image model based on the prompt word information is used as the image to be stored.

[0024] Optionally, encoding the image to be stored to obtain an encoded file includes:

[0025] In the case that the image to be stored is unique, the image to be stored is divided into a plurality of coding units, and the coding units are configured to be in an intra-frame coding mode, so as to process the image to be stored and its corresponding image identification information into the coded file based on the intra-frame coding mode;

[0026] In the case that the image to be stored is not unique, the non-unique image to be stored is numbered, and each image to be stored and its corresponding numbering information are processed into the encoding file based on the intra-frame encoding mode and the inter-frame encoding mode.

[0027] Optionally, processing the to-be-stored image and its corresponding image number information into the encoded file based on the intra-frame encoding mode includes:

[0028] After dividing the image to be stored into a plurality of coding units, obtaining an intra-frame prediction residual of the coding unit in an intra-frame prediction mode, wherein the intra-frame prediction residual is first residual data between the coding unit and an intra-frame prediction image unit;

[0029] The first residual data is transformed and quantized to obtain a quantization coefficient, and the quantization coefficient and the image number information corresponding to the image to be stored are entropy encoded to obtain the encoded file.

[0030] Optionally, processing each to-be-stored image and its corresponding encoding information into the encoding file based on the intra-frame encoding mode and the inter-frame encoding mode includes:

[0031] Acquire an inter-frame prediction residual of a second coding unit in an inter-frame prediction mode, where the second coding unit is a coding unit of a second frame of an image to be stored, the inter-frame prediction residual is second residual data between the second frame of an image to be stored and a first frame of an image to be stored, and the second frame of an image to be stored is later than the first frame of an image to be stored in terms of timing;

[0032] Encoding is performed based on the second residual data, the first encoding file corresponding to the first frame of image to be stored, and the numbering information corresponding to the second frame of image to be stored to obtain a second encoding file of the second frame of image to be stored.

[0033] Optionally, associating the encoded file with the key feature data and storing them in a target storage area of ​​the electronic device includes:

[0034] After obtaining the key feature data of each image to be stored, storing the key feature data of each image to be stored in a first database of the electronic device, wherein the first database is a data set storing the key feature data of the image to be stored;

[0035] storing the encoded file obtained by encoding each image to be stored in a target storage area of ​​the electronic device; and

[0036] Each encoding file is associated with the corresponding key feature data to form an associated file including a mapping relationship between the encoding file and the key feature data.

[0037] Optionally, obtaining a target image from the target storage area in response to a target search requirement includes:

[0038] In response to obtaining a target search requirement, determining target feature data matching the target search requirement from a first database, wherein the first database is a data set storing key feature data of the image to be stored;

[0039] Determining a matching target encoding file from the target storage area based on the target feature data;

[0040] The target encoded file is decoded to obtain the target image.

[0041] An image processing device, comprising:

[0042] An acquisition module, configured to acquire key feature data of the image to be stored in response to obtaining at least one frame of the image to be stored;

[0043] An encoding module, used for encoding the image to be stored to obtain a corresponding encoding file, wherein the data volume of the encoding file is smaller than the data volume of the image to be stored;

[0044] The storage processing module is used to associate the encoding file with the key feature data and store them in a target storage area of ​​the electronic device, so that the electronic device can obtain a target image from the target storage area in response to a target search requirement.

[0045] An electronic device, comprising:

[0046] Memory, used to store computer instruction sets;

[0047] A processor is used to implement the image processing method provided in any one of the above items by executing a computer instruction set.

[0048] A storage medium carries one or more computer instruction sets, and when the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the image processing method as described in any one of the above. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 It is a flowchart of the image processing method provided by this application;

[0051] Figure 2This is a flowchart of encoding an image based on an intra-frame encoding mode provided by the present application;

[0052] Figure 3 is an example of a coding unit and its corresponding reference pixel provided in this application;

[0053] Figure 4 It is a schematic diagram of a pixel value prediction method provided by the present application;

[0054] Figure 5 It is a processing framework diagram of intra-frame coding provided by this application;

[0055] Figure 6 This is a flowchart of encoding an image based on an intra-frame coding mode and an inter-frame coding mode provided by the present application;

[0056] Figure 7 is a flow chart of searching a target image from a target storage area provided by the present application;

[0057] Figure 8 This is the overall processing framework diagram for encoding, storing and searching images provided by this application;

[0058] Fig. 9 It is a structural diagram of the image processing device provided by the present application;

[0059] Fig.10 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] The embodiment of the present application provides an image processing method and device, which is used to solve the problem that the traditional technology has a high storage resource usage and affects the operating rate of the terminal device in the image data storage scenario. The image processing method provided by the embodiment of the present application can be applied to electronic devices in many general or special computing device environments or configurations, such as: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, etc.

[0062] See also Figure 1 As shown in the method flow chart, the image processing method provided in the embodiment of the present application may at least include the following steps 101 to 103, and these steps are described in detail below.

[0063] Step 101: In response to obtaining at least one frame of an image to be stored, key feature data of the image to be stored is obtained.

[0064] The image to be stored may be, but is not limited to, a screenshot image, an image obtained by taking a photo, an image obtained or received by the electronic device from other devices, etc. Among them, the screenshot image may include, but is not limited to, various application screenshot images such as game screenshot images and video screenshot images, or may also be a desktop screenshot image of a non-application interface. The image obtained or received by the electronic device from other devices may include, but is not limited to, images received or downloaded by the electronic device through short-range communication (such as Bluetooth communication) or long-range network communication (such as mobile network) with other devices.

[0065] Accordingly, the electronic device can obtain at least one frame of image to be stored by means of, but not limited to, screenshots, photographs and / or communication with other devices. For example, in a scenario where a target application such as a game is running, screenshots can be taken periodically at set time intervals using a corresponding screenshot tool to obtain multiple frames of game screenshot images to be stored, or a group of photos to be stored in the scene can be obtained based on photographing in the corresponding scenario.

[0066] In response to obtaining at least one frame of image to be stored, the electronic device obtains key feature data of the image to be stored, so as to provide support for generating corresponding index data for the image in a subsequent image storage process.

[0067] The key feature data of the image to be stored can be used to characterize, indicate or describe the image to be stored.

[0068] Optionally, the key feature data of the image to be stored may include any one or more of image feature data of the image area in the image to be stored and / or text feature data of the text area in the image to be stored obtained by identifying the content of the image to be stored, label data for the image to be stored, image description data obtained by converting the image to be stored into text, and image parameter data of the image to be stored. In addition, the key feature data of the image to be stored may also include image annotation data or link data.

[0069] The image parameter data of the image to be stored may include but is not limited to the image size, resolution, generation time, source, etc. of the image to be stored. The image annotation data of the image to be stored may be label information added by the user or device to the image to be stored that can indicate the target object, such as entity labels such as people, animals, plants, food, transportation, and daily items. The link data of the image to be stored may include but is not limited to the website address, QR code, and other information contained in the image to be stored, which is used to indicate but not limited to the source or content of the image to be stored.

[0070] In an optional implementation manner, the key feature data of the image to be stored may specifically include original data information of any one or more of the above information.

[0071] However, not limited to this, in other optional implementations, the original data information of any one or more of the above information of the image to be stored can also be vectorized, and the obtained vector data can be used as key feature data of the image to be stored to further reduce the storage space occupied.

[0072] Step 102: Encode the image to be stored to obtain a corresponding encoded file, wherein the data volume of the encoded file is smaller than the data volume of the image to be stored.

[0073] Among them, when the image to be stored is unique, that is, only one frame of the image to be stored is obtained, the image to be stored is divided into multiple coding units (Coding Unit, CU), and the coding units are configured as intra-frame coding mode, so as to process the image to be stored and its corresponding image identification information into the encoded file based on the intra-frame coding mode.

[0074] Specifically, the image to be stored may be divided into a plurality of pixel blocks, for example, the image to be stored may be divided into square pixel blocks of a size of 25*25 or 16*16, and each pixel block may be used as a coding unit of the image to be stored. Each pixel block includes a plurality of pixels with continuous pixel coordinate positions, for example, 25*25 or 16*16 pixels with continuous pixel coordinate positions.

[0075] The image identification information corresponding to the image to be stored is used to uniquely identify the image to be stored, and may include but is not limited to information such as the number (global number) and name of the image to be stored.

[0076] In the case where the image to be stored is unique, on the basis of dividing the image to be stored into multiple coding units, the corresponding intra-frame coding process can be performed on each coding unit of the image to be stored and the image identification information corresponding to the image to be stored based on the intra-frame coding mode to obtain the coding file corresponding to the image to be stored. The data volume of the coding file is smaller than the data volume of the image to be stored. By performing the intra-frame coding process on the obtained frame of the image to be stored (i.e., the image to be stored is unique), the data of the image to be stored can be effectively compressed, thereby reducing the space and time costs required for its storage and transmission.

[0077] In the case that the image to be stored is not unique, the embodiment of the present application numbers the non-unique images to be stored (for example, assigning a corresponding frame number to each image to be stored based on the timing information between the images to be stored) so as to uniquely identify each image to be stored based on the corresponding numbering information, and process each image to be stored and its corresponding numbering information into the encoded file based on the intra-frame coding mode and the inter-frame coding mode.

[0078] In this case, for the obtained multiple frames of images to be stored, specifically based on the intra-frame coding mode and the inter-frame coding mode, corresponding intra-frame coding processing and inter-frame coding processing can be performed on each image to be stored as needed (for the first frame of the image to be stored in the multiple frames of images to be stored, such as the image to be stored with the earliest time sequence, only intra-frame coding is required, and there is no need to perform inter-frame coding processing), and an encoded file corresponding to each image to be stored is obtained. The data amount of the encoded file corresponding to the image to be stored is smaller than the data amount of the image to be stored. By performing corresponding coding processing on the image to be stored based on the intra-frame coding mode and the inter-frame coding mode, the data of each frame of the image to be stored can be effectively compressed, thereby reducing the space and time costs required for its storage and transmission.

[0079] In the case where the obtained image to be stored is not unique, each frame of the image to be stored can also be divided into multiple coding units based on image segmentation processing, such as into multiple pixel blocks. By performing intra-frame coding and inter-frame coding processing on each coding unit of each frame of the image to be stored as needed, intra-frame coding and inter-frame coding can be realized for each frame of the image to be stored (only intra-frame coding is performed on the first frame).

[0080] Optionally, each encoded file obtained by encoding the at least one frame of the image to be stored can be formed into an encoded video file with an image frame number, and the encoded video file includes image encoding data of each frame of the image to be stored and its corresponding frame number.

[0081] Step 103: Associating the encoding file with the key feature data and storing them in a target storage area of ​​the electronic device, so that the electronic device can obtain a target image from the target storage area in response to a target search requirement.

[0082] After obtaining the key feature data of each image to be stored, the key feature data of each image to be stored may be stored in a first database of the electronic device, where the first database is a data set storing the key data of the image to be stored.

[0083] The first database of the electronic device can be stored in the local storage area of ​​the electronic device, in the storage area of ​​an external device (such as a USB flash drive, a mobile hard disk, etc.), or in a cloud storage area. There is no restriction on this and it can be determined according to the actual application situation.

[0084] For each image to be stored, the encoded file obtained by encoding each image to be stored can be stored in the target storage area of ​​the electronic device, and the encoded file of each image to be stored can be associated with the corresponding key feature data to form an associated file including a mapping relationship between the encoded file and the key feature data.

[0085] The associated file may be implemented in the form of, but not limited to, an index table or a mapping relationship table, and is used as index information for indexing each image to be stored, so that the electronic device can obtain the target image from the target storage area in response to a target search requirement.

[0086] Specifically, but not limited to, the image feature data, text feature data, label data, image parameter data (such as image size, resolution, generation time, source, etc.) and / or key feature data such as image description data obtained by image-to-text processing of each image to be stored can be associated or mapped with the frame number or serial number of the encoded image to obtain an associated file in the form of the index table or mapping relationship table, so that according to the target search needs, the corresponding label or text feature data, image description data and other key feature data can be matched through search processing, and the target image that meets the target search needs can be located in combination with the mapping relationship between the key feature data and the frame number or serial number of the encoded image.

[0087] It is easy to understand that the target image that meets the target search requirements and is located during image search is specifically the encoded file of the target image rather than its original image data. Subsequently, the original image data of the target image can be obtained by decoding the encoded file of the target image in order to perform required applications such as visualization of the target image.

[0088] The target search demand may be, but is not limited to, from a target user, an operating system of an electronic device, or an application installed on the electronic device. Specifically, it may be, but is not limited to, search information input by the target user, the operating system of the electronic device, or an application installed on the electronic device based on their own needs.

[0089] The associated files in the form of index tables or mapping relationship tables can be stored in the target storage area, or in other areas different from the target storage area, and can also be stored in the local storage area of ​​the electronic device, or the storage area of ​​an external device, or in a cloud storage area.

[0090] The target storage area may be a local storage area of ​​the electronic device, a storage area of ​​an external device, or a cloud storage area, and there is no limitation to this. The encoding file and the key feature data may be stored in the same storage area or in different storage areas.

[0091] In summary, the image processing method provided in the embodiment of the present application, for at least one frame of the image to be stored, realizes data compression of the image to be stored by encoding each frame of the image to be stored, reduces its data volume, saves the occupation of data storage space, and avoids affecting the operating rate of the terminal device. And by obtaining the key feature data of the image to be stored, and associating and storing the encoding file of the image to be stored with the key feature data, the electronic device can respond to the target search demand and obtain the target image that meets the target search demand, so that the present application can save the image data storage space while meeting the search demand for the stored image.

[0092] In an optional embodiment, obtaining at least one frame of image to be stored may be implemented by including at least one of the following 1-1 to 1-5:

[0093] 1-1. Monitor the highlight moments of a target player in a target game application, and obtain screenshot images corresponding to the highlight moments as the images to be stored.

[0094] Optionally, the target player may be a player represented by an account used by the device user in the target game application.

[0095] The target game application may be any game application installed on the electronic device, or may be a key game application set by the device user or automatically set by the electronic device based on game features for taking screenshots of the application interface. The game features may include but are not limited to game type, scale, applicable player population, etc.

[0096] The highlight moments of the target player in the target game application may include, but are not limited to, key, important or wonderful operation moments of the target player in the target game application, and / or moments when the target player obtains key achievements based on operation behaviors in the target game application.

[0097] Optionally, when starting and running the target game application, the electronic device can start the corresponding monitoring and screenshot tool by responding to the startup event of the target game application, or start the monitoring and screenshot tool based on the user's manual operation. Thereafter, the tool is used to monitor the highlight moments of the target player in the target game application, and screenshots of the application interface / screen interface are taken when the highlight moments are monitored to obtain the screenshot image corresponding to the highlight moment, which is used as the image to be stored.

[0098] In actual applications, not limited to the above implementations, screenshot tools can also be used to periodically capture the application interface / screen interface of the target game application at set time intervals, and the screenshot images obtained by periodic screenshots can be used as the images to be stored.

[0099] 1-2. In response to a screenshot operation on a current application interface of the electronic device, a screenshot image obtained by the screenshot operation is used as the image to be stored.

[0100] The current application may be a game application, or may be other types of applications such as a video playback application, a live broadcast application, a conference application, etc.

[0101] The screenshot operation for the current application interface of the electronic device may be a manual operation performed by the current user, or may be a screenshot operation automatically performed by the electronic device based on intelligent judgment.

[0102] Among them, the electronic device can, but is not limited to, intelligently determine whether to trigger a screenshot operation based on a time strategy or information features of the current application interface.

[0103] When judging based on the time strategy, it can be optional to specifically determine whether the time interval between the current moment and the most recent historical screenshot moment reaches a set duration, and if so, trigger a screenshot operation; when judging based on the information features of the current application interface, it can be but is not limited to judging whether to trigger a screenshot operation based on, for example, detecting a player's key / important operation in a game application triggers a screenshot operation on the game application; detecting that key / important information is displayed in a conference application triggers a screenshot operation on the conference application, etc.

[0104] 1-3. In response to obtaining an image acquisition operation performed by a target user on an electronic device, an image obtained by the image acquisition operation is used as the image to be stored.

[0105] Optionally, the image acquisition operation may include a photographing operation performed using an electronic device, and the photographs obtained by photographing may be used as the images to be stored. For example, a corresponding scene is photographed using an electronic device to obtain a group of photographs of the scene, and the group of photographs is used as the images to be stored.

[0106] The image acquisition operation acting on the electronic device can be manually triggered by a user or automatically triggered by the electronic device based on relevant applications or policies. For example, the electronic device periodically triggers a photo-taking operation based on a preset time interval to take a photo of the scene.

[0107] 1-4. In response to obtaining an image sharing request from a target device, using the received shared image as the image to be stored, wherein the target device is a device that establishes a target communication connection with the electronic device.

[0108] The target communication connection may be a wired connection or a wireless connection, such as a wired connection established between the electronic device and the target device based on a USB (Universal Serial Bus) connection line, or a wireless connection established between the electronic device and the target device based on Bluetooth or Wi-Fi (mobile hotspot).

[0109] When the target device that establishes a target communication connection with the electronic device shares an image with the electronic device, the electronic device may use the received shared image as the image to be stored.

[0110] 1-5. In response to obtaining prompt word information input into a target raw image model, a generated image generated by the target raw image model based on the prompt word information is used as the image to be stored.

[0111] The target image generation model is a model used to generate images and is a type of generative artificial intelligence model.

[0112] The prompt word information is a prompt input into the target raw image model, and may include but is not limited to descriptive information of the image to be generated by the target raw image model, such as the theme, style, and image characteristics of the image to be generated, so as to guide the target raw image model to generate an image that meets the requirements.

[0113] After the prompt word information is input into the target raw image model, the target raw image model generates an image that meets the requirements based on the prompt word information. In this embodiment, the generated image generated by the target raw image model based on the prompt word information is used as the image to be stored.

[0114] In practical applications, it is not limited to using the target image generation model to generate images. The video generation model can also be used to generate images. The video generation model can be used to generate videos and is also a type of generative artificial intelligence model. Accordingly, in response to obtaining the prompt word information input into the video generation model, the image generated by the video generation model based on the input prompt word information can be used as the image to be stored.

[0115] This embodiment uses the image obtained by at least one of the above methods as the image to be processed to participate in subsequent associated feature data acquisition, encoding and other processing, which can effectively reduce the data volume of the images to be stored obtained by the above methods, save data storage space, and avoid affecting the operating rate of the terminal device, while also meeting the search needs for the stored images.

[0116] In an optional implementation, the step 101 of obtaining the key feature data of the image to be stored may be implemented as the following steps 2-1 to 2-2:

[0117] 2-1. Identify the object content in the image to be stored, where the object content includes text and / or image.

[0118] Before identifying the object content in the image to be stored, the image to be stored may be first segmented according to the object type included. The object type may include but is not limited to image information (such as people, animals, scenery, etc.), text information (such as related description information, annotation information), etc. Based on the object type actually contained in the image to be stored, the image to be stored may be segmented into text areas and / or image areas, but is not limited to.

[0119] On this basis, the segmented text regions and / or image regions can be respectively subjected to object content recognition. Specifically, based on the different object types contained in the different segmented regions, different segmented regions can be subjected to recognition processing using different recognition methods.

[0120] If the segmented area includes a text area, optionally, text recognition based on OCR (Optical Character Recognition) can be performed on the text area to obtain a corresponding text recognition result. The OCR-based text recognition can be based on OCR technology to recognize text strings in the text area, or based on the recognition of text strings, further semantic recognition of text strings is performed to extract key strings, such as extracting keywords in the text content, etc. The text recognition result can correspondingly include all strings or key strings (such as keywords) in the text content.

[0121] If the segmented area includes an image area, optionally, the image content of the image area may be recognized based on an image recognition model to obtain an image recognition result.

[0122] Among them, the image recognition model is an artificial intelligence model used for image recognition. By learning and analyzing a large amount of image data, the image recognition model can automatically identify target objects, scenes, people, etc. in the image, so as to provide multiple functions such as detection and recognition of targets in the image, feature extraction, and scene understanding.

[0123] The image recognition results of the image content in the image area may include but are not limited to image types (such as people, mountains, buildings, animals, objects), identifiers (such as person names, animal categories, building names or categories, object names or categories, etc.), associated descriptions (descriptions that can characterize the current state or type of the image object, such as animals on the grass) and / or key features (such as grass, PC, etc.) or key image parameters (such as color parameters or generation time, source), etc.

[0124] 2-2. Use the recognized text recognition results and / or image recognition results as the key feature data of the image to be stored; or, vectorize the recognized text content and / or image content, and use the obtained vector data as the key feature data of the image to be stored.

[0125] After obtaining text recognition results and / or image recognition results by performing object content recognition on the image to be stored, in an optional embodiment, the obtained text recognition results and / or image recognition results can be directly used as key feature data of the image to be stored. For example, keywords in the text content in the text area of ​​the image to be stored and / or the type, identification, key description, generation time, etc. of the object in the image area can be used as key feature data of the image to be stored.

[0126] In other optional embodiments, the identified text content and / or image content can be further vectorized, for example, the keywords in the text content in the text area can be vectorized to obtain keyword vectors, or the type, identification, key description, generation time, etc. of the object in the image area can be vectorized to obtain vector data of these contents, and the vector data obtained by vectorization can be used as key feature data of the image to be stored.

[0127] This embodiment obtains the key feature data of the image to be stored, and provides a basis for the subsequent construction of an associated file for the mapping relationship between the encoding file and the key feature data used to characterize the image to be stored, so that the electronic device can obtain the target image from the target storage area in response to the target search demand, thereby meeting the search demand for the stored image. In addition, this embodiment can support the user or the device to construct a search condition based on the image area content and / or text area content of the image to search for the desired target image, thereby meeting the user's diverse search needs.

[0128] In an optional implementation, the step of obtaining the key feature data of the image to be stored in step 101 may be further implemented as at least one of the following 3-1 to 3-5:

[0129] 3-1. Identify the image content of the image to be stored, and use the obtained recognition result or feature vector data as key feature data of the image to be stored.

[0130] Optionally, text and / or non-text object content (such as text content or image content) in the image to be stored can be identified based on image region segmentation, and the identification result can be used as key feature data of the image to be stored. For example, text recognition results such as text keywords and / or image recognition results such as object types of objects in the image region, object association data (such as associated descriptions that can characterize the current state or type of the image object) can be used as key feature data of the image to be stored.

[0131] Alternatively, the recognition results of the text and / or non-text object content in the image to be stored are further vectorized, and the feature vector data obtained based on the vectorization processing is used as the key feature data of the image to be stored, so as to reduce the data volume of the key feature data and correspondingly reduce the amount of storage space occupied.

[0132] The feature vector data here may include key feature vectors obtained after vectorization processing of the recognition results, such as feature vectors of facial features, key image parameter features, etc.

[0133] 3-2. Perform image-to-text processing on the image to be stored, and use the target text data in the obtained text description data as key feature data of the image to be stored.

[0134] Optionally, a picture-to-text conversion model may be used to perform picture-to-text conversion on the image to be stored, obtain text description data for describing the overall content of the image to be processed, extract target text data from the text description data, and use the extracted target text data as key feature data of the image to be stored.

[0135] The target text data is the text description data obtained by conversion that can characterize the uniqueness of key image features in the image, such as the name of the object, the type of the object, the object's clothing, the location of the object, etc.

[0136] The image-text conversion model used may be a local image-text conversion model of the electronic device, or may be a cloud-based image-text conversion model. There is no limitation to this and it may depend on the actual application situation.

[0137] 3-3. Obtain image parameter data of the image to be stored, and use the image parameter data as key feature data of the image to be stored.

[0138] The image parameter data of the image to be stored may include, but is not limited to, any one or more parameter data of the image size, resolution, generation time, source, etc. of the image to be stored.

[0139] 3-4. Obtaining operation behavior data on the image to be stored, and using the operation behavior data as key feature data of the image to be stored.

[0140] The operation behavior data for the image to be stored may include but is not limited to editing data, cropping operation data, etc. of the image to be stored, such as editing operation data such as adding text, adding expressions, adding special effects, adding annotation information, or editing operation data such as zooming, beautifying, filtering, adjusting image parameters, or cropping operation data for cropping the background area of ​​the image, etc.

[0141] 3-5. Obtain source data of the image to be stored, and use the source data as key feature data of the image to be stored.

[0142] The source data of the image to be stored may be any form of data that can prove the source of the image, and may specifically include but is not limited to endorsement data such as blockchain data or watermark of the image to be stored.

[0143] This embodiment obtains the key feature data of the image to be stored, and provides a basis for the subsequent construction of an associated file for the mapping relationship between the encoding file and the key feature data used to characterize the image to be stored, so that the electronic device can respond to the target search demand and obtain the target image from the target storage area, thereby meeting the search demand for the stored image. In addition, this embodiment can support the user or device to construct a search condition based on any one of the multiple data such as the content information of the image, the corresponding text description data after the image is converted into text, the image parameter data, the source data and / or the operation behavior data on the image, and search for the desired target image, further meeting the user's diverse search needs.

[0144] In an optional embodiment, when encoding the image to be stored, for the case where the image to be stored is unique, refer to Figure 2 The flowchart shown in the figure, based on the intra-frame coding mode, the implementation process of processing the image to be stored and its corresponding image number information into the coded file may include:

[0145] Step 201: After dividing the image to be stored into a plurality of coding units, obtain an intra-frame prediction residual of the coding unit in an intra-frame prediction mode, wherein the intra-frame prediction residual is first residual data between the coding unit and an intra-frame prediction image unit.

[0146] After the image to be stored is divided into a plurality of coding units, such as a plurality of pixel blocks, an intra-frame prediction residual of each coding unit in an intra-frame prediction mode may be determined.

[0147] For each coding unit, when determining its intra-frame prediction residual in the intra-frame prediction mode, optionally, a reference pixel corresponding to the coding unit in the image to be stored can be first determined, and then pixel values ​​of each pixel in the coding unit are predicted based on the reference pixel corresponding to the coding unit to obtain the predicted pixel value of each pixel in the coding unit, and the difference between the actual pixel value and the predicted pixel value of each pixel in the coding unit is used as the intra-frame prediction residual of the coding unit in the intra-frame prediction mode.

[0148] The intra-frame prediction residual corresponding to the coding unit (i.e., the difference between the actual pixel value and the predicted pixel value of each pixel in the coding unit) can also be understood as the first residual data between the coding unit and its corresponding intra-frame prediction image unit. The intra-frame prediction image unit corresponding to the coding unit is an image unit formed by each predicted pixel value obtained by predicting the pixel value of the coding unit based on the reference pixel corresponding to the coding unit.

[0149] In an optional implementation, the reference pixels corresponding to the coding unit in the image to be stored specifically include each pixel adjacent to the upper boundary of the coding unit and each pixel adjacent to the left boundary of the coding unit in the area outside the coding unit on the image to be stored. In this implementation, each reference pixel corresponding to the coding unit may also be referred to as an upper outer boundary pixel and a left outer boundary pixel of the coding unit.

[0150] However, it is not limited thereto. In another optional implementation, the reference pixels corresponding to the coding unit in the image to be stored include each pixel adjacent to the lower boundary of the coding unit and each pixel adjacent to the right boundary of the coding unit in the area outside the coding unit on the image to be stored. In this implementation, each reference pixel corresponding to the coding unit may also be referred to as the lower outer boundary pixel and the right outer boundary pixel of the coding unit.

[0151] See also Figure 3 Schematic diagram, assuming that the current coding unit is Figure 3 If the pixel block Pixel-block is formed by N*N pixels in the image to be stored, then the reference pixels corresponding to the current coding unit in the image to be stored Image may include each pixel adjacent to the upper boundary of the Pixel-block in the area outside the pixel block Pixel-block on the Image and each pixel adjacent to the left boundary of the Pixel-block, that is, the upper outer boundary pixels and the left outer boundary pixels corresponding to the pixel block Pixel-block, and the number of pixels is 2N+1. Of course, the reference pixels corresponding to the current coding unit in the image to be stored Image may also include each pixel adjacent to the lower boundary of the Pixel-block in the area outside the pixel block Pixel-block on the Image and each pixel adjacent to the right boundary of the Pixel-block, that is, the lower outer boundary pixels and the right outer boundary pixels corresponding to the pixel block Pixel-block, and the number of pixels is also 2N+1.

[0152] For multiple coding units into which the image to be stored is divided, there are cases where some coding units lack at least some reference pixels in the image to be stored. For example, for the above-mentioned first implementation of the reference pixels, when the coding unit is located at the left boundary and / or the upper boundary of the image, it will lack at least some reference pixels in the image; for the above-mentioned second implementation of the reference pixels, when the coding unit is located at the lower boundary and / or the right boundary of the image, it will lack at least some reference pixels in the image.

[0153] For a coding unit that is missing at least part of the reference pixels in the image to be processed, pixel values ​​of the missing at least part of the reference pixels can be filled based on a preset pixel value filling method to obtain pixel values ​​of the missing at least part of the reference pixels.

[0154] Optionally, when only some pixels are missing from the 2*N+1 reference pixels corresponding to the coding unit, the pixel values ​​of the missing pixels may be determined and filled based on the pixel values ​​of the neighboring pixels of the missing pixels. For example, the average pixel value of each neighboring pixel of the pixels may be determined, and the average pixel value may be used as the pixel value of the pixels.

[0155] When all 2*N+1 reference pixels corresponding to the coding unit are missing, the pixel median (ie, the median of 0 to 256 pixel values, 128) may be used to fill in the pixel values ​​of the missing reference pixels.

[0156] After determining the reference pixel corresponding to the coding unit in the image to be stored, the pixel value prediction of each pixel in the coding unit can be further performed based on the reference pixel corresponding to the coding unit. Optionally, for each pixel to be predicted in the coding unit, a first reference pixel having the same horizontal coordinate as the pixel to be predicted and a second reference pixel having the same vertical coordinate as the pixel to be predicted can be determined from the reference pixels corresponding to the coding unit, and the pixel value of the pixel to be predicted is predicted based on the pixel value of the first reference pixel and the pixel value of the second reference pixel.

[0157] See also Figure 4 The pixel value prediction method provided is an example, assuming that the coding unit is Figure 4 A pixel block formed by N*N pixels in the pixel block, and assuming that the 2*N+1 reference pixels of the pixel block are the left outer boundary pixels and the upper outer boundary pixels corresponding to the pixel block, then for each pixel to be predicted in the pixel block, optionally, its pixel value can be predicted based on the following formula:

[0158] Px=w1*P1+w2*P2;

[0159] Wherein, Px is the predicted pixel value of the pixel x to be predicted in the pixel block, P1 is the pixel value of the first reference pixel with the same horizontal coordinate as the pixel x to be predicted among the reference pixels corresponding to the pixel block, and P2 is the pixel value of the second reference pixel with the same vertical coordinate as the pixel x to be predicted among the reference pixels corresponding to the pixel block. w1 and w2 are weights corresponding to the first reference pixel and the second reference pixel, respectively, wherein the weights w1 and w2 are verified specific values ​​that can make the prediction result meet the expected accuracy.

[0160] For example, for Figure 4 The pixel point c in the image can be specifically predicted based on the pixel value Pa of the first reference pixel a with the same horizontal coordinate and the pixel value Pb of the second reference pixel b with the same vertical coordinate. The calculation formula used is: Pc=w1*Pa+w2*Pb.

[0161] On this basis, for each coding unit of the image to be stored, the actual pixel value and the predicted pixel value of each pixel in the coding unit can be further matched one by one according to the pixel position, and the corresponding actual pixel value and the predicted pixel value can be differenced to obtain the difference (residual) between the actual pixel value and the predicted pixel value of each pixel in the coding unit, and then the difference between the actual pixel value and the predicted pixel value of each pixel in the coding unit is used as the intra-frame prediction residual of the coding unit in the intra-frame prediction mode, that is, the first residual data.

[0162] Step 202: transform and quantize the first residual data to obtain a quantization coefficient, and perform entropy coding on the quantization coefficient and the image number information corresponding to the image to be stored to obtain the coded file.

[0163] After the intra-frame prediction residual corresponding to each coding unit, that is, the first residual data, is obtained, the first residual data corresponding to each coding unit is transformed and quantized.

[0164] It is easy to understand that the intra-frame prediction residual corresponding to each coding unit is also the first residual data, which is specifically a residual matrix formed by the residual values ​​(the difference between the actual pixel value and the predicted pixel value) corresponding to each pixel in the coding unit.

[0165] When the first residual data corresponding to each coding unit is transformed, the first residual data in the form of a residual matrix can be subjected to DCT (discrete cosine transform), that is, the residual matrix of the first residual data is multiplied by a DCT transform matrix on the left and right to obtain a coefficient block of the same size as the residual matrix, and the absolute value of each coefficient in the coefficient block represents the energy at a certain frequency. Since the energy of an image is concentrated in low frequencies in reality, the absolute value of the coefficient is generally smaller in the coefficient block as it is farther away from the upper left corner.

[0166] After transforming the first residual data corresponding to each coding unit to obtain the corresponding coefficient block, the coefficient block corresponding to each coding unit is further quantized. Optionally, the coefficient block can be divided by a quantization value or a quantization matrix, and then the division result is rounded to an integer to quantize the coefficient block and obtain a quantized coefficient. Among them, most of the small coefficients far away from the upper left corner will be quantized to 0, 1, 2, -1, -2... Such small numbers suitable for compression.

[0167] It is worth mentioning that quantization will cause distortion. The larger the divisor used in the division operation during quantization, the higher the compression rate, but the higher the degree of distortion. In practical applications, the compression rate and the degree of distortion can be comprehensively considered to determine the divisor used in the division operation during quantization.

[0168] On this basis, the quantization coefficient corresponding to each coding unit and the image number information or image frame number information corresponding to the image to be stored can be entropy encoded to obtain the encoded file corresponding to the image to be stored. In addition, the pixel values ​​of the reference pixels corresponding to each coding unit (such as the 2*N+1 left boundary pixels and upper boundary pixels, or the 2*N+1 right boundary pixels and lower boundary pixels corresponding to the coding unit) need to be encoded.

[0169] When encoding the above information, Run Length Coding (RLC) or Variable-length Codes (VLC) may be used but is not limited to the use of such techniques.

[0170] Among them, RLC technology will skip the 0 value when encoding, and only encode the number of consecutive 0s, so as to achieve data compression effect and reduce data storage volume. VLC technology assigns shorter codes to smaller numbers. Since the probability of small numbers appearing is greater than that of large numbers, VLC can also achieve data compression effect.

[0171] Optional, see Figure 5In the intra-frame coding flow chart shown, before transforming the first residual data (residual matrix) corresponding to each coding unit, the first residual data can also be firstly subjected to median subtraction processing, wherein each residual value in the first residual data, that is, each residual value in the residual matrix, can be subtracted from the pixel median 128, and the value range of each residual value in the residual matrix is ​​0 to 255. After subtracting the pixel median 128, the value range becomes -128 to 127, thereby achieving the effect of reducing the data volume of the first residual data and saving storage space. On this basis, the first residual data corresponding to each coding unit after the median subtraction processing can be further subjected to subsequent transformation, quantization, entropy coding and other processing to obtain the encoded file corresponding to the image to be stored.

[0172] The encoding file corresponding to the image to be stored includes encoding data obtained by encoding pixel values ​​of reference pixels corresponding to each encoding unit of the image to be stored, quantization coefficients, and serial number or frame number information of the image to be stored.

[0173] Subsequently, the encoding file of the image to be stored may be associated with the key feature data of the image to be stored and stored in a target storage area of ​​the electronic device, so that the electronic device can obtain the target image from the target storage area in response to a target search requirement.

[0174] In summary, for the case where the obtained image to be stored is unique, this embodiment achieves compression of the data of the image to be stored from the perspective of intra-frame coding by determining the intra-frame prediction residual of the coding unit based on the intra-frame coding mode, effectively reducing the amount of data of the image to be stored, saving the storage space occupied, and avoiding the impact on the operating rate of the terminal device. In addition, the above-mentioned pixel value prediction method provided in this embodiment can use 2*N+1 reference pixels of the left / upper outer boundary or the right / lower outer boundary of the coding unit to predict the pixel values ​​of N*N pixels in the coding unit, while the H265 standard intra-frame prediction method requires 4*N+1 reference pixels to predict the pixel values ​​of N*N pixels in the coding unit. Therefore, compared with the H265 standard intra-frame prediction method, the embodiment of the present application reduces the complexity of the prediction process, and further reduces the data storage amount of the image, saving storage space.

[0175] In an optional embodiment, in the case where the image to be stored is not unique, after the non-unique image to be stored is numbered, refer to Figure 6 The process of processing each to-be-stored image and its corresponding numbering information into the coded file based on the intra-frame coding mode and the inter-frame coding mode may include:

[0176] Step 601, obtain the inter-frame prediction residual of the second coding unit in the inter-frame prediction mode, the second coding unit is the coding unit of the second frame image to be stored, the inter-frame prediction residual is the second residual data between the second frame image to be stored and the first frame image to be stored, and the second frame image to be stored is later than the first frame image to be stored in time sequence.

[0177] In the case where the obtained image to be stored is not unique, optionally, a plurality of frames of images to be stored may be firstly sorted based on a preset sorting method to obtain a sequence of images to be stored.

[0178] The preset sorting method is used to improve the similarity between images that meet the proximity relationship after sorting compared to before sorting, and can be specifically sorted based on the order of image generation time based on the timeline, or sorted based on the similarity between different images, etc. For example, multiple screenshot images are sorted based on the timeline in the order of screenshot time, or multiple photos collected by the electronic device are sorted based on the content similarity between the photos (such as arranging similar pictures taken in the same scene, the same time period, and the same place adjacently).

[0179] Different images satisfying the proximity relationship may mean that different images are adjacent to each other, or the number of images spaced between different images does not reach a set number.

[0180] After obtaining a sequence of images to be stored by sorting, for each image to be stored in the sequence of images to be stored, the target type of the image to be stored can be determined according to the corresponding position of the image to be stored in the sequence of images to be stored, and then according to the target type of the image to be stored, the specific encoding processing to be performed on the image to be stored in the intra-frame encoding mode and the inter-frame encoding mode can be determined.

[0181] The target type of the image to be stored is a first type or a second type, the first type indicates that the image to be stored has no corresponding reference frame in the image sequence to be stored, and the second type indicates that the image to be stored has a corresponding reference frame in the image sequence to be stored.

[0182] If the target type of the image to be stored is the first type, in the intra-frame coding mode and the inter-frame coding mode, only the intra-frame coding process needs to be performed on the image to be stored. On the contrary, if the target type of the image to be stored is the second type, in the intra-frame coding mode and the inter-frame coding mode, both the intra-frame coding process and the inter-frame coding process need to be performed on the image to be stored.

[0183] In an optional embodiment, for the sequence of images to be stored, the target type of the first frame of the image to be stored can be directly determined as the first type, and the target type of each non-first frame of the image to be stored can be determined as the second type. The reference frame corresponding to each non-first frame of the image to be stored is preferably the previous adjacent frame image corresponding to the non-first frame of the image to be stored in the sequence of images to be stored. That is to say, in this embodiment, each frame of the image to be stored other than the first frame is inter-frame encoded with its corresponding previous adjacent frame image in the sequence of images to be stored as a reference frame. Since each image to be stored does not have the continuity feature between different video frames in a real video file in time, the previous adjacent frame of the image to be stored is preferentially used as the reference frame corresponding to the image to be stored. This is more valuable for reference. When performing inter-frame encoding on the image to be stored, the effect of minimizing the inter-frame difference between the image to be stored and its reference frame can be achieved, which is conducive to further improving the data compression rate.

[0184] In another embodiment, the sequence of images to be stored may be divided into multiple groups, each group including a certain number of frames of images to be stored, and the images to be stored in the same group are continuous in the sequence of images to be stored. On this basis, the target type of the first frame of the image to be stored in each group is determined as the first type, and the target type of the non-first frame of the images to be stored in the group is determined as the second type, and the reference frame corresponding to each non-first frame of the images to be stored in the same group is the first frame of the image to be stored in the group.

[0185] In another embodiment, for the sequence of images to be stored, the target type of the first frame of the image to be stored can be directly determined as the first type. For the current image to be stored that is not the first frame, it is determined whether the first data amount and the second data amount meet the proximity condition. If so, it is determined that the target type of the current image to be stored is the second type. If not, it is determined that the target type of the current image to be stored is the first type.

[0186] The proximity condition may be set as that a difference between the first data amount and the second data amount is smaller than a threshold.

[0187] The first data amount is the data amount of intra-frame coded data of the current image to be stored, and the second data amount is the data amount of intra-frame coded data of the previous adjacent frame image corresponding to the current image to be stored in the sequence of images to be stored. In this implementation, if the target type of the current image to be stored is the second type, the reference frame corresponding to the current image to be stored is the previous adjacent frame image corresponding to the current image to be stored.

[0188] The intra-frame coded data of the image to be stored includes coded data obtained by encoding each coding unit and the corresponding reference pixel of the image to be stored based on the intra-frame coding mode.

[0189] As described above, if the target type of the image to be stored is the first type, in the intra-frame coding mode and the inter-frame coding mode, only the intra-frame coding process needs to be performed on the image to be stored. For the intra-frame coding process performed on the image to be stored, please refer to the relevant description of the intra-frame coding process in the previous embodiment, which will not be repeated here.

[0190] If the target type of the image to be stored is the second type, in the intra-frame coding mode and the inter-frame coding mode, both the intra-frame coding process and the inter-frame coding process need to be performed on the image to be stored. When performing the inter-frame coding process on the image to be stored, the image to be stored can be used as the second frame of the image to be stored, and the reference frame corresponding to the image to be stored can be used as the first frame of the image to be stored, and the inter-frame prediction residual of each second coding unit of the second frame of the image to be stored in the inter-frame prediction mode is determined, and the inter-frame prediction residual is the second residual data between the second frame of the image to be stored and the first frame of the image to be stored, specifically the difference data between the coding data of each second coding unit of the second frame of the image to be stored and the coding data of each first coding unit of the first frame of the image to be stored.

[0191] Step 602: Perform encoding processing based on the second residual data, the first encoding file corresponding to the first frame of image to be stored, and the numbering information corresponding to the second frame of image to be stored to obtain a second encoding file of the second frame of image to be stored.

[0192] Based on step 601, encoding processing can be performed based on the second residual data, the first encoded file corresponding to the first frame of image to be stored, and the numbering information corresponding to the second frame of image to be stored, and the encoding result is used as the second encoded file of the second frame of image to be stored.

[0193] The second frame of the image to be stored is specifically an image to be stored whose corresponding target type in the sequence of images to be stored is the second type.

[0194] It is easy to understand that for the images to be stored whose corresponding target type in the image sequence to be stored is the first type, since only intra-frame coding processing needs to be performed on them in the intra-frame coding mode and the inter-frame coding mode, the encoding result obtained by encoding them based on the intra-frame coding processing can be directly used as their encoded files.

[0195] Subsequently, the encoding file of each image to be stored may be associated with the key feature data of the image to be stored and stored in a target storage area of ​​the electronic device, so that the electronic device can obtain the target image from the target storage area in response to a target search requirement.

[0196] To summarize, for the case where the obtained images to be stored are not unique, this embodiment processes each image to be stored and its corresponding numbering information into the encoded file based on the intra-frame coding mode and the inter-frame coding mode, thereby compressing the data of the image to be stored from an intra-frame perspective based on the intra-frame prediction method, and compressing the data of the image to be stored from an inter-frame perspective based on the similarity / overlapping parts between the inter-frame image data, thereby effectively reducing the data amount of the image to be stored, saving the occupancy of data storage space, and avoiding affecting the operating rate of the terminal device.

[0197] In an alternative embodiment, see Figure 7 In the image processing method provided in the present application, obtaining the target image from the target storage area in response to the target search requirement can be implemented as follows:

[0198] Step 701: In response to obtaining a target search requirement, determine target feature data matching the target search requirement from a first database, where the first database is a data set storing key feature data of the image to be stored.

[0199] The target search demand may be, but is not limited to, from a target user, an operating system of an electronic device, or an application installed on the electronic device. Specifically, it may be, but is not limited to, search information input by the target user, the operating system of the electronic device, or an application installed on the electronic device based on their own needs.

[0200] The target user, the operating system of the electronic device, or the application installed on the electronic device can construct search conditions based on, but not limited to, the image object content to be searched, the corresponding text description data after the image is converted into text, the image parameter data, the source data, and / or the operation behavior on the image, so as to characterize its target search needs based on the search conditions.

[0201] In response to obtaining the target search requirement, the electronic device can specifically determine the target feature data that matches the search conditions in the target search requirement from the key feature data of each image stored in the first database, for example, determine the target feature data that matches the keywords (image content or text keywords) in the search conditions, etc.

[0202] Step 702: Determine a matching target encoding file from the target storage area based on the target feature data.

[0203] Based on step 701, the image number or frame number corresponding to the target feature data can be further determined based on the constructed and stored associated files, such as an index table or a mapping relationship table, and then the corresponding target encoding file can be located from the target storage area according to the determined image number or frame number.

[0204] Step 703: Decode the target encoded file to obtain the target image.

[0205] After obtaining the target encoded file, the target encoded file can be decoded to obtain its corresponding target image, and the decoded target image is returned to the requesting party such as the user or device, so that the requesting party can perform the desired application on the target image, such as distinguishing and perceiving the user scene based on the image content of the target image.

[0206] The decoding process of the target encoded file is the inverse process of the encoding process of the stored image.

[0207] Among them, if the target encoding file is an encoding file obtained by performing only intra-frame encoding processing on the corresponding image to be stored, the pixel value of each pixel in each encoding unit can be predicted based on the reference pixel value of each encoding unit contained in the target encoding file, and the predicted pixel value of each pixel in each encoding unit is summed with the residual value represented / corresponding to the corresponding quantization coefficient to restore the actual pixel value of each pixel in each encoding unit, and then restore the image corresponding to the target encoding file based on each encoding unit.

[0208] If the target coding file is a coding file obtained by performing intra-frame coding and inter-frame coding processing on the corresponding image to be stored, first, based on the second residual data contained in the target coding file and the first coding file corresponding to the first frame of the image to be stored, the intra-frame coding data of the second frame of the image to be stored corresponding to the target coding file can be restored, and then based on the above-mentioned pixel value prediction and the summation of the predicted pixel value and the corresponding residual value, the image corresponding to the target coding file can be restored.

[0209] See also Figure 8 , provides an overall processing framework diagram for encoding, storing and searching the images to be stored in the present application, which includes obtaining the key feature data of the images to be stored, image encoding, storage and image search. The processing process of each link can be specifically referred to the description of the corresponding embodiments of the present application, which will not be repeated here.

[0210] In this embodiment, the electronic device can obtain the target image from the target storage area in response to the target search demand, thereby meeting the user's image search needs; and this embodiment supports the user or device to construct search conditions based on any one or more of the image content information, the corresponding text description data after the image is converted into text, the image parameter data, the source data and / or the operation behavior data on the image, thereby meeting the user's diverse search needs.

[0211] Corresponding to the above-mentioned image processing method, the embodiment of the present application further provides an image processing device, whose composition structure is as follows: Fig. 9 As shown, including:

[0212] An acquisition module 901 is used for acquiring key feature data of the image to be stored in response to obtaining at least one frame of the image to be stored;

[0213] The encoding module 902 is used to encode the image to be stored to obtain a corresponding encoding file, wherein the data volume of the encoding file is smaller than the data volume of the image to be stored;

[0214] The storage processing module 903 is used to associate the encoding file with the key feature data and store them in a target storage area of ​​the electronic device, so that the electronic device can obtain a target image from the target storage area in response to a target search requirement.

[0215] In an alternative embodiment,

[0216] In an optional implementation manner, the acquisition module 901, when acquiring the key feature data of the image to be stored, is specifically used to:

[0217] Identifying object content in the image to be stored, wherein the object content includes text and / or image;

[0218] Using the identified text recognition result and / or image recognition result as key feature data of the image to be stored; or,

[0219] The recognized text content and / or image content is vectorized, and the obtained vector data is used as key feature data of the image to be stored.

[0220] In an optional implementation manner, the acquisition module 901, when, is specifically configured to perform at least one of the following:

[0221] Identify the image content of the image to be stored, and use the obtained identification result or feature vector data as key feature data of the image to be stored;

[0222] Performing image-to-text processing on the image to be stored, and using target text data in the obtained text description data as key feature data of the image to be stored;

[0223] Obtaining image parameter data of the image to be stored, and using the image parameter data as key feature data of the image to be stored;

[0224] Obtaining operation behavior data on the image to be stored, and using the operation behavior data as key feature data of the image to be stored;

[0225] The source data of the image to be stored is obtained, and the source data is used as the key feature data of the image to be stored.

[0226] In an optional implementation, the acquisition module 901 is further used to obtain at least one frame of image to be stored;

[0227] When obtaining at least one frame of image to be stored, the acquisition module 901 is specifically configured to perform at least one of the following:

[0228] Monitoring the highlight moments of a target player in a target game application, and obtaining a screenshot image corresponding to the highlight moment as the image to be stored;

[0229] In response to a screenshot operation on a current application interface of the electronic device, taking a screenshot image obtained by the screenshot operation as the image to be stored;

[0230] In response to obtaining an image acquisition operation performed by a target user on an electronic device, using an image obtained by the image acquisition operation as the image to be stored;

[0231] In response to obtaining an image sharing request from a target device, using the received shared image as the image to be stored, wherein the target device is a device that establishes a target communication connection with the electronic device;

[0232] In response to obtaining the prompt word information input into the target raw image model, a generated image generated by the target raw image model based on the prompt word information is used as the image to be stored.

[0233] In an optional implementation manner, the encoding module 902 is specifically configured to:

[0234] In the case that the image to be stored is unique, the image to be stored is divided into a plurality of coding units, and the coding units are configured to be in an intra-frame coding mode, so as to process the image to be stored and its corresponding image identification information into the coded file based on the intra-frame coding mode;

[0235] In the case that the image to be stored is not unique, the non-unique image to be stored is numbered, and each image to be stored and its corresponding numbering information are processed into the encoding file based on the intra-frame encoding mode and the inter-frame encoding mode.

[0236] In an optional implementation manner, when the encoding module 902 processes the to-be-stored image and its corresponding image number information into the encoded file based on the intra-frame encoding mode, it is specifically configured to:

[0237] After dividing the image to be stored into a plurality of coding units, obtaining an intra-frame prediction residual of the coding unit in an intra-frame prediction mode, wherein the intra-frame prediction residual is first residual data between the coding unit and an intra-frame prediction image unit;

[0238] The first residual data is transformed and quantized to obtain a quantization coefficient, and the quantization coefficient and the image number information corresponding to the image to be stored are entropy encoded to obtain the encoded file.

[0239] In an optional implementation manner, when the encoding module 902 processes each to-be-stored image and its corresponding encoding information into the encoded file based on the intra-frame encoding mode and the inter-frame encoding mode, it is specifically configured to:

[0240] Acquire an inter-frame prediction residual of a second coding unit in an inter-frame prediction mode, where the second coding unit is a coding unit of a second frame of an image to be stored, the inter-frame prediction residual is second residual data between the second frame of an image to be stored and a first frame of an image to be stored, and the second frame of an image to be stored is later than the first frame of an image to be stored in terms of timing;

[0241] Encoding is performed based on the second residual data, the first encoding file corresponding to the first frame of image to be stored, and the numbering information corresponding to the second frame of image to be stored to obtain a second encoding file of the second frame of image to be stored.

[0242] In an optional implementation manner, the storage processing module 903 is specifically configured to:

[0243] After obtaining the key feature data of each image to be stored, storing the key feature data of each image to be stored in a first database of the electronic device, wherein the first database is a data set storing the key feature data of the image to be stored;

[0244] storing the encoded file obtained by encoding each image to be stored in a target storage area of ​​the electronic device; and

[0245] Each encoding file is associated with the corresponding key feature data to form an associated file including a mapping relationship between the encoding file and the key feature data.

[0246] In an optional implementation manner, the above device further includes a search module, which is used to:

[0247] In response to obtaining a target search requirement, target feature data matching the target search requirement is determined from a first database, wherein the first database is a data set storing key feature data of the image to be stored; a matching target encoding file is determined from the target storage area based on the target feature data; and the target encoding file is decoded to obtain the target image.

[0248] The present application also discloses an electronic device, the composition structure of the electronic device, such as Fig.10 As shown, including at least:

[0249] A memory 10, for storing a computer instruction set;

[0250] A set of computer instructions can be implemented in the form of a computer program.

[0251] The processor 20 is used to implement the image processing method provided by any of the above method embodiments by executing a computer instruction set.

[0252] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU) or other programmable logic devices, etc.

[0253] The electronic device includes a display device and / or includes a display interface and can be connected to an external display device.

[0254] Optionally, the electronic device also includes a camera assembly, and / or is connected to an external camera assembly.

[0255] In addition, the electronic device may also include components such as a communication interface and a communication bus. The memory, the processor and the communication interface communicate with each other through the communication bus.

[0256] The communication interface is used for communication between electronic devices and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0257] An embodiment of the present application also provides a storage medium, which carries one or more computer instruction sets. When the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the image processing method as described in any one of the above items.

[0258] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0259] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0260] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or the part that makes a creative contribution can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0261] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0262] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, comprising: In response to obtaining at least one frame of image to be stored, acquiring key feature data of the image to be stored; Performing encoding processing on the image to be stored to obtain a corresponding encoded file, wherein the data volume of the encoded file is smaller than the data volume of the image to be stored; The encoding file is associated with the key feature data and stored in a target storage area of ​​the electronic device, so that the electronic device can obtain a target image from the target storage area in response to a target search requirement.

2. The method according to claim 1, wherein: Acquiring key feature data of the image to be stored, including: Identifying object content in the image to be stored, wherein the object content includes text and / or image; Using the identified text recognition result and / or image recognition result as key feature data of the image to be stored; or, The recognized text content and / or image content is vectorized, and the obtained vector data is used as key feature data of the image to be stored.

3. The method according to claim 1, wherein: Acquiring key feature data of the image to be stored includes at least one of the following: Identify the image content of the image to be stored, and use the obtained identification result or feature vector data as key feature data of the image to be stored; Performing image-to-text processing on the image to be stored, and using target text data in the obtained text description data as key feature data of the image to be stored; Obtaining image parameter data of the image to be stored, and using the image parameter data as key feature data of the image to be stored; Obtaining operation behavior data on the image to be stored, and using the operation behavior data as key feature data of the image to be stored; The source data of the image to be stored is obtained, and the source data is used as the key feature data of the image to be stored.

4. The method according to any one of claims 1 to 3, wherein: Obtaining at least one frame of image to be stored includes at least one of the following: Monitoring the highlight moments of a target player in a target game application, and obtaining a screenshot image corresponding to the highlight moment as the image to be stored; In response to a screenshot operation on a current application interface of the electronic device, taking a screenshot image obtained by the screenshot operation as the image to be stored; In response to obtaining an image acquisition operation performed by a target user on an electronic device, using an image obtained by the image acquisition operation as the image to be stored; In response to obtaining an image sharing request from a target device, using the received shared image as the image to be stored, wherein the target device is a device that establishes a target communication connection with the electronic device; In response to obtaining the prompt word information input into the target raw image model, a generated image generated by the target raw image model based on the prompt word information is used as the image to be stored.

5. The method according to claim 1, wherein: The image to be stored is encoded to obtain an encoded file, including: In the case that the image to be stored is unique, the image to be stored is divided into a plurality of coding units, and the coding units are configured to be in an intra-frame coding mode, so as to process the image to be stored and its corresponding image identification information into the coded file based on the intra-frame coding mode; In the case that the image to be stored is not unique, the non-unique image to be stored is numbered, and each image to be stored and its corresponding numbering information are processed into the encoding file based on the intra-frame encoding mode and the inter-frame encoding mode.

6. The method according to claim 5, wherein: Processing the to-be-stored image and its corresponding image number information into the encoded file based on the intra-frame encoding mode includes: After dividing the image to be stored into a plurality of coding units, obtaining an intra-frame prediction residual of the coding unit in an intra-frame prediction mode, wherein the intra-frame prediction residual is first residual data between the coding unit and an intra-frame prediction image unit; The first residual data is transformed and quantized to obtain a quantization coefficient, and the quantization coefficient and the image number information corresponding to the image to be stored are entropy encoded to obtain the encoded file.

7. The method according to claim 5 or 6, wherein: Processing each to-be-stored image and its corresponding encoding information into the encoding file based on the intra-frame encoding mode and the inter-frame encoding mode includes: Acquire an inter-frame prediction residual of a second coding unit in an inter-frame prediction mode, where the second coding unit is a coding unit of a second frame of an image to be stored, the inter-frame prediction residual is second residual data between the second frame of an image to be stored and a first frame of an image to be stored, and the second frame of an image to be stored is later than the first frame of an image to be stored in terms of timing; Encoding is performed based on the second residual data, the first encoding file corresponding to the first frame of image to be stored, and the numbering information corresponding to the second frame of image to be stored to obtain a second encoding file of the second frame of image to be stored.

8. The method according to claim 1, wherein: Associating the encoded file with the key feature data and storing them in a target storage area of ​​the electronic device, comprising: After obtaining the key feature data of each image to be stored, storing the key feature data of each image to be stored in a first database of the electronic device, wherein the first database is a data set storing the key feature data of the image to be stored; storing the encoded file obtained by encoding each image to be stored in a target storage area of ​​the electronic device; and Each encoding file is associated with the corresponding key feature data to form an associated file including a mapping relationship between the encoding file and the key feature data.

9. The method according to claim 1, wherein: Responding to a target search requirement, obtaining a target image from the target storage area comprises: In response to obtaining a target search requirement, determining target feature data matching the target search requirement from a first database, wherein the first database is a data set storing key feature data of the image to be stored; Determining a matching target encoding file from the target storage area based on the target feature data; The target encoded file is decoded to obtain the target image.

10. An image processing device, comprising: An acquisition module, configured to acquire key feature data of the image to be stored in response to obtaining at least one frame of the image to be stored; An encoding module, used for encoding the image to be stored to obtain a corresponding encoding file, wherein the data volume of the encoding file is smaller than the data volume of the image to be stored; The storage processing module is used to associate the encoding file with the key feature data and store them in a target storage area of ​​the electronic device, so that the electronic device can obtain a target image from the target storage area in response to a target search requirement.

Images