Image processing method and device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method and apparatus, an electronic device, and a storage medium. The method comprises: obtaining a simulation library based on an interface provided by at least one function of a hardware resource; processing image coding information of an image to be processed obtained by the hardware resource through the simulation library to obtain an image processing result; and storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information. According to the image processing method of an embodiment of the present disclosure, the hardware resource can be simulated through the interface of the hardware resource function to obtain a simulation library of the hardware resource, thereby simulating the function of the hardware resource of the terminal device on a general computing platform. It can also facilitate the development and debugging of the hardware resource function and other business logic based on the image processing result, thereby improving the efficiency of development and debugging.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] Currently, artificial intelligence technology is widely used in fields such as security and surveillance. As chip computing power continues to be moved to the edge, smart cameras, as a typical example of this edge-side computing, are becoming increasingly powerful and widely used. For example, smart cameras are a common edge-side device. Smart cameras typically use embedded systems. Chip manufacturers provide SDKs (software development kits) corresponding to these embedded systems, and users write code based on the interfaces provided by the chip SDK to perform corresponding functions.

[0003] Developing and debugging embedded systems for edge devices like these smart cameras is complex, and the devices themselves are expensive and limited in quantity. Furthermore, embedded development often suffers from efficiency issues (cross-compilation, on-board debugging, and troubleshooting tools are limited), making it less convenient than general-purpose computing platforms like x86. However, general-purpose computing platforms lack the hardware capabilities of edge devices. Summary of the Invention

[0004] The present disclosure provides an image processing method and device, an electronic device, and a storage medium.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: obtaining a simulation library of at least one function of a hardware resource according to an interface provided by the at least one function, the simulation library comprising a functional module for simulating at least one function of the hardware resource; processing image coding information of an image to be processed acquired by the hardware resource through the simulation library to obtain an image processing result; and storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

[0006] According to the image processing method of the embodiment of the present invention, the hardware resources can be simulated through the functional interface of the hardware resources to obtain a simulation library of the hardware resources, so that the functions of the hardware resources of the terminal device can be simulated on the general computing platform to perform corresponding processing on the image and obtain the image processing results. This not only enables the general computing platform to simulate the hardware resources, but also facilitates the development and debugging of the functions of the hardware resources and other business logic according to the image processing results, which can improve the efficiency of development and debugging.

[0007] In one possible implementation, based on the interface of at least one function of the hardware resources of the end-side device, a simulation library of the at least one function is obtained, including: reading the interface to obtain declaration information of the at least one function; performing empty implementation on the declaration information to obtain an empty implementation result of the at least one function; determining a target function in the at least one function based on the empty implementation result; and establishing a functional module in the simulation library based on the declaration information of the target function, wherein the functional module is used to implement the target function.

[0008] In this way, the target function to be simulated can be determined through empty implementation, and simulation can be performed based on the declaration information to obtain functional modules and build a simulation library, so that the general computing platform can simulate the functions of the end-side device, facilitating development and debugging.

[0009] In a possible implementation, the method further includes: decompressing the image to be processed through the simulation library to obtain image coding information of the image to be processed.

[0010] In this way, image coding information that is convenient for image processing can be obtained through the functional modules in the simulation library, thereby improving decompression efficiency.

[0011] In a possible implementation manner, the method further includes: storing the image processing result in header information corresponding to the image identifier of the image to be processed.

[0012] In this way, the image processing results can be stored in a unified header information, thereby improving the convenience of searching and calling.

[0013] In a possible implementation, the method further includes: searching the result coding information for an image processing result corresponding to the function according to the name of the function.

[0014] In a possible implementation, the method further includes: searching the result coding information for an image processing result corresponding to the function according to the name of the function.

[0015] In a possible implementation, the method further includes: encoding the result encoding information to obtain push information; and pushing the push information to a terminal device for display.

[0016] In this way, push information that is easy for users to obtain can be obtained, making it easy for users to use the result coding information.

[0017] In a possible implementation manner, the preset field of the image coding information includes a tail field of the image coding information.

[0018] In this way, since the image processing results are uniformly stored in the tail field and the image processing results have a preset naming method, the image processing results can be searched based on the name of the processing or function performed, thereby improving the convenience of searching and calling.

[0019] According to one aspect of the present disclosure, an image processing device is provided, including: a simulation library acquisition module for obtaining a simulation library of at least one function of a hardware resource based on an interface provided by the at least one function of the hardware resource, the simulation library including a functional module for simulating at least one function of the hardware resource; a processing module for processing image coding information of an image to be processed obtained by the hardware resource through the simulation library to obtain an image processing result; and a first storage module for storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

[0020] In one possible implementation, the simulation library acquisition module is further used to: read the interface to obtain declaration information of at least one function; perform empty implementation on the declaration information to obtain an empty implementation result of at least one function; determine a target function in the at least one function based on the empty implementation result; establish a functional module in the simulation library based on the declaration information of the target function, and the functional module is used to implement the target function.

[0021] In a possible implementation, the apparatus further includes: a decompression module, configured to decompress the image to be processed through the simulation library to obtain image coding information of the image to be processed.

[0022] In a possible implementation, the apparatus further includes: a second storage module, configured to store the image processing result in header information corresponding to the image identifier of the image to be processed.

[0023] In a possible implementation, the apparatus further includes: a search module configured to search, according to the name of the function, the result coding information for an image processing result corresponding to the function.

[0024] In a possible implementation, the apparatus further includes: a push module configured to encode the result encoding information to obtain push information; and push the push information to a terminal device for display.

[0025] In a possible implementation manner, the preset field of the image coding information includes a tail field of the image coding information.

[0026] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0027] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0028] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0030] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure is shown;

[0031] Figure 2 A schematic diagram illustrating a simulation library according to an embodiment of the present disclosure is shown;

[0032] Figure 3 A schematic diagram showing header information according to an embodiment of the present disclosure;

[0033] Figure 4 A schematic diagram illustrating a preset information format of a tail field of image coding information according to an embodiment of the present disclosure;

[0034] Figure 5 A schematic diagram illustrating an application of an image processing method according to an embodiment of the present disclosure is shown;

[0035] Figure 6 A block diagram showing an image processing apparatus according to an embodiment of the present disclosure;

[0036] Figure 7 A block diagram illustrating an electronic device according to an embodiment of the present disclosure is shown;

[0037] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0041] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0042] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the image processing method includes:

[0043] In step S11, according to an interface provided by at least one function of the hardware resource, a simulation library of the at least one function is obtained, wherein the simulation library includes a functional module for simulating the at least one function of the hardware resource;

[0044] In step S12, the image coding information of the image to be processed acquired by the hardware resource is processed by the simulation library to obtain an image processing result;

[0045] In step S13, the image processing result is stored in a preset field of the image coding information according to a preset information format to obtain result coding information.

[0046] According to the image processing method of the embodiment of the present invention, the hardware resources can be simulated through the functional interface of the hardware resources to obtain a simulation library of the hardware resources, so that the functions of the hardware resources of the terminal device can be simulated on the general computing platform to perform corresponding processing on the image and obtain the image processing results. This not only enables the general computing platform to simulate the hardware resources, but also facilitates the development and debugging of the functions of the hardware resources and other business logic according to the image processing results, which can improve the efficiency of development and debugging.

[0047] In one possible implementation, the image processing method can be executed by an electronic device such as a terminal device or a server, and the electronic device can serve as the above-mentioned computing platform. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in a memory. Alternatively, the method can be executed by a server.

[0048] In one possible implementation, the hardware resources may include the hardware resources of the end-side device, and a general-purpose computing platform (e.g., an x86 platform) can be used to simulate the hardware resources of an end-side device, such as a smart camera, such as a GPU / NPU / DSP / NNIE. The code related to the hardware resources of an end-side device, such as a smart camera, often only accounts for a small portion of the total code (e.g., typically less than 5%), while pure CPU code unrelated to the hardware resources often accounts for the vast majority of the total code (e.g., typically more than 95%). Most complex business logic code occurs in pure CPU code, which has characteristics such as complex logic, prone to errors, and prone to code corruption requiring reconstruction. Therefore, development and debugging through the end-side device's own embedded system would be very complex. However, using a general-purpose computing platform for development and debugging would greatly reduce the difficulty of development and debugging and improve efficiency. Therefore, using a computing platform to simulate the functions related to the hardware resources of the end-side device (e.g., simulating the functions implemented by the aforementioned small portion of code) allows all the functions of the end-side device to be implemented on a general-purpose computing platform, which is of great significance for improving development and debugging efficiency.

[0049] Figure 2 A schematic diagram of a simulation library according to an embodiment of the present disclosure is shown as follows: Figure 2As shown, in the end-side devices (for example, smart cameras), embedded systems are usually used. The chip manufacturers of embedded systems can usually provide SDK libraries (for example, libChipSdk.so libraries). Developers can develop based on the SDK interface in the SDK library, for example, write code, and the developed functions (for example, application software) can call the functions or operators in the SDK through the above interface, thereby calling the hardware resources of the end-side device for calculation. However, as mentioned above, its development and debugging process is very complicated. Therefore, by simulating the above SDK library on a general computing platform (for example, x86 platform), a simulation library (for example, libMockSdk.so library) can be obtained, so that various functions can be developed and debugged on the general computing platform. The developed functions (for example, application software) can call the interface in the simulation library, thereby calling the CPU of the general computing platform for calculation. Thus, the purpose of development and debugging on a general computing platform can be achieved, so as to reduce the difficulty of development and debugging and improve the efficiency of development and debugging.

[0050] In one possible implementation, in step S11, a simulation library of hardware resources can be obtained based on the interface provided by at least one function of the hardware resource. For example, the function (for example, the above-mentioned SDK) that can call the hardware resource of the terminal device is simulated to obtain a functional module that can simulate the function, and the library composed of functional modules corresponding to multiple functions is determined as a simulation library. The simulation library can be used to simulate the SDK library composed of multiple SDKs of the terminal device. The functional modules in the simulation library can simulate the functions of the SDK, but the computing resources that can be called are the CPU of the general computing platform, rather than the hardware resources of the terminal device that can be called by the SDK, thereby achieving the purpose of developing and debugging various functions (for example, application software) developed based on the functional modules on the general computing platform, so as to reduce the difficulty of development and debugging and improve the efficiency of development and debugging.

[0051] In one possible implementation, step S11 may include: reading the interface to obtain declaration information of at least one function; performing empty implementation on the declaration information to obtain an empty implementation result of at least one function; determining a target function in the at least one function based on the empty implementation result; and establishing a functional module in the simulation library based on the declaration information of the target function, wherein the functional module is used to implement the target function.

[0052] In one possible implementation, the functional modules in the simulation library can be generated to obtain the simulation library. The functional modules can be generated based on the corresponding SDK interfaces. As described above, the SDK interface is used to call functions or operators within the SDK, thereby enabling development using the SDK's functionality. Therefore, the SDK interface can be used to express the SDK's functionality. Furthermore, based on the information read from the interface, the expressed functionality can be simulated to obtain the functional modules.

[0053] In one possible implementation, an information reading tool can be used to read the SDK interface. For example, the declaration information of a function or operator in the SDK can be read. For example, the declaration information may include the function or operator's functionality, logic, and related variables. This disclosure does not limit the content of the declaration information. In this example, the readelf information reading tool can be used to read the SDK interface and obtain the declaration information of the function or operator in the SDK.

[0054] In a possible implementation, after obtaining the declaration information, an empty implementation can be performed based on the declaration information. For example, a parameterless constructor can be generated, the object in the constructor is empty, and its function is mainly to return an empty implementation result based on the declaration information, for example, based on the function, property, category and other information of the function described in the declaration function, the empty implementation result is returned. For example, if the SDK function output is a pointer, the empty implementation result returned by the empty implementation is NULL, if the SDK function output is integer data, the empty implementation result returned by the empty implementation is 0, if the SDK function is a typeless function (void function), the empty implementation result returned by the empty implementation is empty, or no result is returned. In the example, the empty implementation tool obtained by the open source parser (ANTLR) and a specific version of the function grammar (for example, the grammar version grammars-v4 of the C language) can be used to perform the above-mentioned empty implementation to obtain an empty implementation result. After performing the above-mentioned empty implementation, a constructor is obtained in a general computing platform, so that it can be successfully compiled in the general computing platform.

[0055] In one possible implementation, the target function to be simulated can be selected based on the empty implementation results of various functions (for example, SDK). In the example, if the SDK is used to create or close resources, it does not need to use the hardware resources of the end-side device, so there is no need to simulate it. Through the above-mentioned empty implementation processing, a construction function with an empty object can be obtained to simulate the function of the function. If the SDK needs to use the hardware resources of the end-side device, it needs to be simulated. The empty implementation result returned by the empty implementation of such a function is usually not empty. Therefore, the function (SDK) with a non-empty empty implementation result can be determined as the target function, and the target function can be simulated to obtain the functional module in the simulation library.

[0056] In one possible implementation, based on the declaration information of the target function, a function with consistent functionality can be called in the function library of the general computing platform. For example, for an SDK with image processing functionality, the corresponding functional module can be obtained based on the function in the libjpeg library. The function module can execute functions that are consistent with the target function and can also provide multiple interfaces, for example, an interface that is consistent with the functional interface of the end-side device.

[0057] In this way, the target function to be simulated can be determined through empty implementation, and simulation can be performed based on the declaration information to obtain functional modules and build a simulation library, so that the general computing platform can simulate the functions of the end-side device, facilitating development and debugging.

[0058] In one possible implementation, taking a smart camera as an example, the image processing process typically includes image acquisition (i.e., capturing images or video frames), decompression, and a series of detection, recognition, and other processing. These processes can all be implemented through the above simulation library. On an end-side device (e.g., a smart camera), developers can use the various functional interfaces of the end-side device to develop image processing functions. For example, they can write a function that can call various functions of the end-side device through the various functional interfaces of the end-side device, thereby utilizing the hardware resources of the end-side device to perform image decompression, detection, recognition, and other processing. Correspondingly, on a general-purpose computing platform, developers can also use the interfaces of various functional modules in the simulation library to develop image processing functions. For example, they can write a function that can call functional modules through the interfaces of various functional modules, thereby utilizing the CPU of the general-purpose computing platform to perform image decompression, detection, recognition, and other processing.

[0059] In one possible implementation, the method further includes: decompressing the image to be processed through the simulation library to obtain image coding information of the image to be processed. In the example, the image to be processed is an image taken by an end-side device (for example, a smart camera). The image to be processed can be a single image taken or a video frame in a video stream. The present disclosure does not limit the type of image to be processed. In the example, the image to be processed can be an image in jpeg format, which can be decompressed by the corresponding functional module in the simulation library to obtain image coding information, for example, YUV information.

[0060] In this way, image coding information that is convenient for image processing can be obtained through the functional modules in the simulation library, thereby improving decompression efficiency.

[0061] In one possible implementation, in step S12, the image coding information may be further processed based on the simulation library, such as detection, recognition, and the like as described above. Step S12 may include performing at least one of target detection, target attribute extraction, target feature extraction, and target feature comparison on the image coding information using the simulation library to obtain the image processing result.

[0062] In one possible implementation, the simulation library may include multiple functional modules, such as a target detection module, an attribute extraction module, a feature extraction module, and a feature comparison module, which can be used to perform target detection, target attribute extraction, target feature extraction, and target feature comparison processing on a general-purpose computing platform. The interfaces of the above functional modules can be called to call at least one of the above functional modules, so as to utilize the CPU of the general-purpose computing platform to perform at least one of target detection, target attribute extraction, target feature extraction, and target feature comparison processing on the image, thereby obtaining image processing results of various processing, such as obtaining at least one of target detection results, attribute extraction results, feature extraction results, and feature comparison results.

[0063] In one possible implementation, the above-mentioned multiple image processing results can be saved, and the functional modules for obtaining the image processing results can be stored correspondingly to the corresponding image processing results. Moreover, when storing, they can also correspond to the image identifier (e.g., frame number, etc.) of the image to be processed. For example, the image processing results can be stored in the format of "frame number-functional module name". In the example, 100_detect.data can represent the target detection result obtained by the target detection module performing target detection processing on the image coding information of the 100th frame image. 200_attribute.data can represent the attribute extraction result obtained by the attribute extraction module performing target attribute extraction processing on the image coding information of the 200th frame image, etc. In the example, not every image needs to be processed by all functional modules, and not every functional module needs to process all images. For example, if a frame image undergoes target detection processing and obtains a target detection result, but does not undergo attribute extraction processing, then the frame image does not have an attribute extraction result.

[0064] In one possible implementation, the method further includes storing the image processing result in header information corresponding to the image identifier of the image to be processed. Various image processing results obtained for each frame of image can be stored in the header information corresponding to that frame of image. For example, if the image encoding information of 100 frames of image undergoes both target detection and target attribute extraction, the target detection result 100_detect.data and the attribute extraction result 100_attribute.data can be stored in the header information jpegapp1 corresponding to the 100th frame of image. This disclosure does not limit the type of header information.

[0065] Figure 3 A schematic diagram showing header information according to an embodiment of the present disclosure is shown. In this example, the image processing results of each frame image can be represented as Figure 3 The format shown is stored in the header information jpegapp1 corresponding to the image identifier (eg, frame number) of each frame, so that the standardized image processing result can be obtained, which facilitates the search and call of the image processing result.

[0066] In this way, the image processing results can be stored in a unified header information, thereby improving the convenience of searching and calling.

[0067] In one possible implementation, as the processing progresses, the header information corresponding to each frame may store image processing results. For example, the header information jpegapp1 corresponding to the 100th frame image stores the target detection result 100_detect.data and the attribute extraction result 100_attribute.data. The header information jpegapp1 corresponding to other frame images may also store multiple image processing results. After the processing is completed, for example, after all images in a batch have completed the required processing, all required image processing results corresponding to each image can be obtained and stored in the header information corresponding to each frame. In step S13, the image processing results can be stored in a preset field of the image coding information according to a preset information format. For example, the image processing results stored in the header information corresponding to each frame are stored in a preset field of the image coding information of each frame to obtain result coding information, that is, image coding information storing the image processing results.

[0068] In one possible implementation, the preset fields of the image coding information include a tail field of the image coding information. Therefore, the image processing results stored in the header information corresponding to each frame can be offset into the preset fields of the image coding information of each frame according to a preset information format to obtain the result coding information.

[0069] Figure 4A schematic diagram showing a preset information format of the tail field of the image coding information according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the preset information format can be consistent with the format stored in the header information. Therefore, when storing the image processing result in the tail field of the image encoding information (for example, YUV), there is no need to change the storage format, only to offset to the tail field.

[0070] In one possible implementation, after obtaining the result coding information, push information can also be obtained based on the result coding information, that is, information that the user can use (for example, can be displayed, or can perform further calculations), so that it can be pushed to the user. The method also includes: encoding the result coding information to obtain push information; pushing the push information to the terminal device for display. In the example, the result coding information is image coding information (for example, YUV) with the image processing result stored in the tail field. The result coding information can be further encoded, for example, JPEG encoding is performed to obtain push information in image format. For example, the frame number can be used as the name of the push information, such as 100.jpeg, 101.jpeg, etc., which respectively represent the push information of the 100th frame and the push information of the 101st frame. The push information in the image format can represent the image processing result. For example, the image processing result of a frame image may include the target detection result of the target object in the image, and the detection result such as the detection box or segmentation box of the target can be included in the push information in the image format. The present disclosure does not limit the format of the push information.

[0071] In this way, push information that is easy for users to obtain can be obtained, making it easy for users to use the result coding information.

[0072] In a possible implementation, according to the name of the function, the image processing result corresponding to the function is searched in the result coding information. When querying the image processing result, the query can be made in the result coding information, and the result coding information includes the image processing results obtained by various functional modules. For example, it can include at least one of the image processing results such as the target detection result obtained by the target detection module, the attribute extraction result obtained by the attribute extraction module, the feature extraction result obtained by the feature extraction module, and the feature comparison result obtained by the feature comparison module. When querying the image processing result of a certain frame of image, the query can be made in its result coding information. The query method may include querying by using the name of the functional module. For example, when querying the target detection result of the 100th frame of image, the target detection result obtained by the target detection module can be queried by using the "detect" keyword in the result coding information of the 100th frame of image. For example, information such as the position coordinates of the detection box can be obtained. The present disclosure does not limit the query method.

[0073] In this way, since the image processing results are uniformly stored in the tail field and the image processing results have a preset naming method, the image processing results can be searched based on the name of the processing or function performed, thereby improving the convenience of searching and calling.

[0074] According to the image processing method of the embodiment of the present disclosure, the target function that needs to be simulated can be determined through empty implementation, and simulation can be performed based on the declaration information to obtain the functional module and build a simulation library, so that the function of the hardware resources can be simulated on the general computing platform to perform corresponding processing on the image and obtain the image processing result. This not only enables the general computing platform to simulate the hardware resources, but also facilitates the development and debugging of the functions of the hardware resources and other business logic according to the image processing results, which can improve the efficiency of development and debugging.

[0075] Figure 5 FIG. 1 shows an application diagram of the image processing method according to an embodiment of the present disclosure, as shown in FIG. Figure 5 As shown, the terminal device may include a smart camera, which may include multiple functions. In the process of developing and debugging application software based on these functions, development and debugging on the smart camera is difficult and inefficient. Therefore, some functions on the smart camera that require calling the hardware resources of the smart camera can be simulated on a general computing platform such as x86 to obtain a simulation library.

[0076] like Figure 5 As shown, the simulation library may include an object detection module, an attribute extraction module, a feature extraction module, and a feature comparison module. These modules, i.e., modules for intelligent computing, can simulate the corresponding functions of a smart camera to process the image encoding information (YUV). During processing, the CPU resources of a general-purpose computing platform can be used to obtain image processing results, such as the object detection result obtained by the object detection module, the attribute extraction result obtained by the attribute extraction module, the feature extraction result obtained by the feature extraction module, and the feature comparison result obtained by the feature comparison module. The obtained image processing results are stored in the header information jpegapp1 corresponding to the image frame number.

[0077] In one possible implementation, after the processing process is completed, the image processing result stored in the header information can be offset to the tail field of the image coding information (YUV) to obtain the result coding information, and the result coding information can be encoded to obtain push information in image format, which can be pushed to the user's terminal for display or further processing, so that the user can debug and develop new functions based on the push information.

[0078] In one possible implementation, the image processing method can be used to simulate specific functions in end-side devices such as smart cameras on a general-purpose computing platform. This allows the CPU resources of the general-purpose computing platform to simulate the hardware resources of the end-side device to process the specific functions. This enables the development and debugging of application software for the end-side device to be performed on the general-purpose computing platform, thereby reducing the difficulty of development and debugging and improving the efficiency of development and debugging. This disclosure does not limit the application areas of the image processing method.

[0079] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0080] In addition, the present disclosure also provides an image processing device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0081] Figure 6 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown. Figure 6 As shown, the device includes: a simulation library acquisition module 11, which is used to obtain a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function of the hardware resource, and the simulation library includes a functional module for simulating at least one function of the hardware resource; a processing module 12, which is used to process the image coding information of the image to be processed obtained by the hardware resource through the simulation library to obtain an image processing result; a first storage module 13, which is used to store the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information.

[0082] In one possible implementation, the simulation library acquisition module is further used to: read the interface to obtain declaration information of at least one function; perform empty implementation on the declaration information to obtain an empty implementation result of at least one function; determine a target function in the at least one function based on the empty implementation result; establish a functional module in the simulation library based on the declaration information of the target function, and the functional module is used to implement the target function.

[0083] In a possible implementation, the apparatus further includes: a decompression module, configured to decompress the image to be processed through the simulation library to obtain image coding information of the image to be processed.

[0084] In a possible implementation, the apparatus further includes: a second storage module, configured to store the image processing result in header information corresponding to the image identifier of the image to be processed.

[0085] In a possible implementation, the apparatus further includes: a search module configured to search, according to the name of the function, the result coding information for an image processing result corresponding to the function.

[0086] In a possible implementation, the apparatus further includes: a push module configured to encode the result encoding information to obtain push information; and push the push information to a terminal device for display.

[0087] In a possible implementation manner, the preset field of the image coding information includes a tail field of the image coding information.

[0088] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0089] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0090] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0091] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0092] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0093] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0094] Figure 7A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal device.

[0095] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0096] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0097] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0098] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0099] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0100] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0101] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0102] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0103] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as wireless network (Wi-Fi), second generation mobile communication technology (2G), third generation mobile communication technology (3G), fourth generation mobile communication technology (4G), long term evolution (LTE) of universal mobile communication technology, fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0104] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0105] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0106] The present disclosure relates to the field of augmented reality. By acquiring image information of a target object in a real-world environment, the relevant features, states, and attributes of the target object are detected or identified using various vision-related algorithms, thereby achieving an AR effect that combines virtual and real life and matches the specific application. For example, the target object may be a face, limbs, gestures, movements, etc. related to the human body, or an identifier or marker related to an object, or a sandbox, display area, or display items related to a venue or location. Vision-related algorithms may involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and object pose or depth detection. Specific applications can involve not only interactive scenarios such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effects display, and virtual model display. Detection or identification of the relevant features, states, and attributes of the target object can be achieved using a convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0107] Figure 8 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0108] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0109] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0110] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0111] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0112] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0113] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0114] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0115] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0116] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0117] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0118] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0119] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0120] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0121] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0122] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: include: Obtaining, according to an interface provided by at least one function of the hardware resource, a simulation library of the at least one function, the simulation library comprising a functional module for simulating the at least one function of the hardware resource; Processing the image coding information of the image to be processed acquired by the hardware resource through the simulation library to obtain an image processing result; storing the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information; The hardware resources include the hardware resources of the end-side device, and all functions of the end-side device are implemented by simulating the hardware resources of the end-side device through a general computing platform. The simulation library is used to simulate an SDK library composed of multiple SDKs of the end-side device, and any functional module in the simulation library is used to simulate the function of one SDK. The step of obtaining a simulation library of at least one function according to an interface provided by at least one function of the hardware resource comprises: Read the interface to obtain declaration information of at least one function; Performing empty implementation on the declaration information to obtain an empty implementation result of at least one function; determining a target function among the at least one function according to the empty realization result; Establishing a function module in the simulation library according to the declaration information of the target function, wherein the function module is used to realize the target function; The step of processing the image coding information of the image to be processed obtained by the hardware resource through the simulation library to obtain an image processing result includes: Decompressing the image to be processed by the simulation library to obtain image coding information of the image to be processed; performing at least one of target detection, target attribute extraction, target feature extraction, and target feature comparison on the image coding information to obtain the image processing result; Storing the image processing result in header information corresponding to the image identifier of the image to be processed; The preset information format is consistent with the format in which the image processing result is stored in the header information of the image identifier of the image to be processed, and the preset field of the image coding information includes the tail field of the image coding information.

2. The method according to claim 1, characterized in that The method further comprises: According to the name of the function, the image processing result corresponding to the function is searched in the result coding information.

3. The method according to claim 1, characterized in that The method further comprises: Encoding the result encoding information to obtain push information; The push information is pushed to the terminal device for display.

4. An image processing device, characterized in that include: a simulation library obtaining module, configured to obtain a simulation library of at least one function of the hardware resource according to an interface provided by the at least one function of the hardware resource, wherein the simulation library includes a function module for simulating the at least one function of the hardware resource; a processing module, configured to process the image coding information of the image to be processed acquired by the hardware resource through the simulation library to obtain an image processing result; A first storage module is configured to store the image processing result in a preset field of the image coding information according to a preset information format to obtain result coding information; The hardware resources include the hardware resources of the end-side device, and all functions of the end-side device are implemented by simulating the hardware resources of the end-side device through a general computing platform. The simulation library is used to simulate an SDK library composed of multiple SDKs of the end-side device, and any functional module in the simulation library is used to simulate the function of one SDK. The simulation library obtains a module, specifically used for: Read the interface to obtain declaration information of at least one function; Performing empty implementation on the declaration information to obtain an empty implementation result of at least one function; determining a target function among the at least one function according to the empty realization result; Establishing a function module in the simulation library according to the declaration information of the target function, wherein the function module is used to realize the target function; The processing module is specifically used to: Decompressing the image to be processed by the simulation library to obtain image coding information of the image to be processed; performing at least one of target detection, target attribute extraction, target feature extraction, and target feature comparison on the image coding information to obtain the image processing result; Storing the image processing result in header information corresponding to the image identifier of the image to be processed; The preset information format is consistent with the format in which the image processing result is stored in the header information of the image identifier of the image to be processed, and the preset field of the image coding information includes the tail field of the image coding information.

5. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images