Image processing method and device, storage medium and electronic device

Abstract

The embodiment of the present application provides an image processing method and device, a storage medium and an electronic device. The method includes: receiving an image processing instruction carrying a first image processing parameter and a second image processing parameter; determining at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models according to the first image processing parameter; performing object recognition on a target image according to the at least one target image processing model to obtain at least one model recognition result; performing a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to an image processing method and device, a storage medium, and an electronic device. Background Art

[0002] With the rapid development of hardware technology, more and more deep vision models are being applied in actual production and life scenarios.

[0003] In actual production and life scenarios, deep vision models need to first use tools such as TensorRT and OpenVINO to convert the model weights trained by neural network frameworks such as PyTorch and TensorFlow into a format that can be deployed through the C++ language; then build an intelligent visual inspection system to load the model file suitable for the deployment environment into the system, perform intelligent inspection, obtain structured information in the image, and perform further processing, such as detecting area intrusions or illegal parking of vehicles.

[0004] However, in the above existing methods, if a certain visual detection model is selected, the intelligent detection model of the system will be limited to this one and cannot be replaced. This is because the hyperparameters of the backbone networks of different models (such as the number of hidden layers, the number of convolution kernels, and the input / output size of the convolution kernels) are not the same; these differences cannot be compatible. Furthermore, if the model needs to be replaced, it means that the system needs to be redeveloped based on the new model. In other words, in the image processing process, the existing technology has the technical problem of low flexibility in model replacement.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide an image processing method and device, a storage medium, and an electronic device to at least solve the technical problem of low flexibility in model replacement existing in the related art.

[0007] According to one embodiment of the present application, an image processing method is provided, including: receiving an image processing instruction carrying a first image processing parameter and a second image processing parameter; determining, based on the first image processing parameter, at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models; performing object recognition on a target image based on the at least one target image processing model to obtain at least one model recognition result; and performing a target operation on the at least one model recognition result based on the operation logic indicated by the second image processing parameter to obtain a target image processing result.

[0008] According to another aspect of the embodiment of the present application, there is also provided an image processing device, including:

[0009] A receiving unit, configured to receive an image processing instruction carrying a first image processing parameter and a second image processing parameter;

[0010] a determining unit, configured to determine, based on the first image processing parameter, at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models;

[0011] A recognition unit, configured to perform object recognition on a target image according to at least one target image processing model to obtain at least one model recognition result;

[0012] The execution unit is used to perform a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

[0013] Optionally, the above-mentioned execution unit includes: an acquisition module, used to obtain an operation function indicated by the second image processing parameter; and a reference operation module, used to perform a reference operation on at least one target operation according to the operation logic indicated by the operation function to obtain a target image processing result.

[0014] Optionally, the above-mentioned determination unit includes: a template parsing module, used to obtain a target configuration file indicated by the first image processing parameter, and parse the meta-algorithm template class; a model determination module, used to determine at least one reference image processing model from multiple candidate image processing models based on at least one model type parameter in the model description parameter set carried in the parsing result; an initialization module, used to initialize at least one reference image processing model to at least one target image processing model based on at least one model description parameter in the model description parameter set, wherein the at least one model description parameter includes at least one of the following: an input parameter identification parameter, an output parameter identification parameter, the number of convolution layers, the number of convolution kernels, and the size of the convolution kernel.

[0015] Optionally, the above-mentioned image processing device also includes: a normalization unit, which is used to: perform object recognition on the target image according to at least one target image processing model, and before obtaining at least one model recognition result, it also includes: determining the image processing function carried in the analysis result; performing image format conversion on the target image according to the image processing function to obtain a first image in a target format, wherein the target format is an image format that matches the target image processing model; and normalizing the pixel values ​​of multiple pixels in the first image to obtain a second image to be input into the target image processing model.

[0016] Optionally, the above-mentioned recognition unit also includes: a type determination module, which is used to determine the target object type according to the key label parameters when the parsing results carry the key label parameters; a result determination module, which is used to determine at least one model recognition result according to the object recognition result that matches the target object type when the recognition result of object recognition performed by at least one target image processing model on the target image includes an object recognition result that matches the target object type.

[0017] Optionally, the above-mentioned execution unit is also used to, when the model recognition result is used to indicate the identified target road object, determine at least one target result image based on the positional relationship between the target road object and the target image area; and perform image rendering operation on the model inference image to obtain the corresponding target result image.

[0018] Optionally, the execution unit is further used to, when a first model recognition result for indicating a first image area where a first road object is located and a second model recognition result for indicating a second image area where a second road object is located are obtained, determine the object position relationship between the first road object and the second road object according to the image position relationship between the first image area and the second image area; and when a plurality of first model recognition results corresponding to a plurality of adjacent frame images for indicating a first image area where a first road object is located and a plurality of second model recognition results corresponding to a plurality of adjacent frame images for indicating a second road object is located are obtained, determine the relative motion relationship between the first road object and the second road object according to the image position relationship between the first image area and the second image area in the plurality of adjacent frame images.

[0019] Optionally, the above-mentioned image processing device also includes a loop unit, which is used to obtain the next image object as the target image according to the running function carried by the image processing instruction; perform image processing on the target image according to the first image processing parameter and the second image processing parameter to obtain the target image processing result; when the current processing result meets the loop stop condition, output the target image processing results corresponding to each of the multiple image objects.

[0020] Optionally, the above-mentioned image processing device also includes a creation unit, which is used to create a target abstract class corresponding to at least one candidate image processing model, wherein the target abstract class is used to load the corresponding candidate image processing model in the target application; and create a target configuration file that matches each of the at least one candidate image processing models, wherein the target configuration file is used to describe the information of the candidate image processing model.

[0021] Optionally, the creation unit is further used to: create a target model loading function of a target abstract class that matches the candidate image processing model according to the model structure of the candidate image processing model, wherein the target model loading function is used to parse the model parameters of the candidate image processing model, and the model parameters include the input parameters, output parameters, the number of convolution layers of the feature extraction network of the candidate image processing model, the number of convolution kernels of the feature extraction network, and the size of each convolution kernel of the feature extraction network; create a label loading function of a target abstract class that matches the candidate image processing model according to the target training label of the candidate image processing model, wherein the target training label is the training set label of the candidate image processing model in the model training stage, and the label loading function is used to convert the target training label into a dictionary type, and the dictionary type is used to describe the target training label with textual meaning; Support image formats, create an image conversion function of the target abstract class that matches the candidate image processing model, wherein the image conversion function is used to convert the image format of the image to be inferred into the image format supported by the candidate image processing model, and perform pixel value normalization operations on all pixels in the image to be inferred after the format conversion to obtain a reference image to be inferred; create an inference function of the target abstract class that matches the candidate image processing model, wherein the inference function is used to perform inference operations on the reference image to be inferred, and output current inference information based on the result of the inference operation, and the current inference information is used to describe the position information of the inspected object in the image to be inferred; create an inference post-processing function of the target abstract class that matches the candidate image processing model, wherein the inference post-processing function is used to perform data parsing operations on the current inference information, and obtain the coordinate value of the inspected object in the image to be inferred based on the result of the data parsing operation.

[0022] Optionally, the above-mentioned creation unit is also used to: create a name reference field, wherein the name reference field is used to describe the model name of the candidate image processing model; create a type reference field, wherein the type reference field is used to describe the model type of the candidate image processing model; create a label path reference field, wherein the label path reference field is used to describe the file path of the label file of the candidate image processing model on the current inference device; create a model path reference field, wherein the model path reference field is used to describe the file path of the candidate image processing model on the current inference device; create a target detection field, wherein the target detection field is used to indicate the object type of at least one inspected object from multiple label types.

[0023] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above image processing method.

[0024] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the image processing method through the computer program.

[0025] Through the present application, an image processing instruction carrying a first image processing parameter and a second image processing parameter is first received; then, based on the first image processing parameter, at least one target image processing model indicated by the image processing instruction is determined from multiple candidate image processing models; then, object recognition is performed on the target image according to the at least one target image processing model to obtain at least one model recognition result; finally, according to the operation logic indicated by the second image processing parameter, a target operation is performed on the at least one model recognition result to obtain a target image processing result. Therefore, in the process of using the image processing model for image processing, when the image processing model needs to be replaced, there is no need to re-develop the model, thereby solving the technical problem of low flexibility in model replacement existing in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 It is a hardware structure block diagram of a server device of an image processing method according to an embodiment of the present application;

[0028] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of an image processing method according to an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of another image processing method according to an embodiment of the present application;

[0031] Figure 5 is a flowchart of another image processing method according to an embodiment of the present application;

[0032] Figure 6is a structural schematic diagram of an image processing device according to an embodiment of the present application;

[0033] Figure 7 It is a structural schematic diagram of an image processing electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of an image processing method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0037] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to an image processing method in an embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0039] As an optional implementation, Figure 2 As shown, the above image processing method includes:

[0040] S202, receiving an image processing instruction carrying a first image processing parameter and a second image processing parameter;

[0041] S204, determining at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models according to the first image processing parameter;

[0042] S206, performing object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result;

[0043] S207, performing a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

[0044] First, step S202 is executed to receive an image processing instruction carrying a first image processing parameter and a second image processing parameter. Then, after receiving the first image processing parameter and the second image processing parameter, step S204 is executed. It should be noted that, in the above step S204, at least one target image processing model indicated by the image processing instruction is determined from a plurality of candidate image processing models according to the first image processing parameter. Specifically, the following steps may be included:

[0045] S204-1, obtaining a target configuration file indicated by the first image processing parameter, and parsing the meta-algorithm template class;

[0046] S204-2, determining at least one reference image processing model from a plurality of candidate image processing models according to at least one model type parameter in the model description parameter set carried in the parsing result;

[0047] S204-3, initialize at least one reference image processing model to at least one target image processing model according to at least one model description parameter in the model description parameter set, wherein the at least one model description parameter includes at least one of the following: an input parameter identification parameter, an output parameter identification parameter, the number of convolution layers, the number of convolution kernels, and the size of the convolution kernel.

[0048] It should be noted that the above meta-algorithm template class takes the abstract class corresponding to the image processing model as a parameter. Furthermore, each image processing model corresponds to an abstract class. The meta-algorithm template class determines its own template type, that is, determines the corresponding image processing model, at runtime.

[0049] It can be understood that the above-mentioned first image processing parameter is used to describe the target configuration file indicated by the image processing instruction, and the above-mentioned target configuration file is used to determine the model name, model type, model path, label path and label name of the image processing model.

[0050] Specifically, assuming that the constructor of the meta-algorithm template class is MetaAlgo(Config&conf), when receiving the first image processing parameter, the content of the parameter conf is parsed. It should be noted that the content of the parameter conf is the model content of a certain image processing model in the target configuration file. In other words, the above parsing operation is to determine multiple model type parameters of a certain image processing model, such as model name, model type, model path, label path and label name.

[0051] For example, the parsing result can be: model type: yoloV5, model name: yoloV5.2.2, model path: / home / username / Document, label path: / home / username / Document, label name: person.

[0052] Further, execute the above step S204-3 to determine at least one reference image processing model from multiple candidate image processing models according to at least one model type parameter in the model description parameter set carried in the parsing result. For example, the reference image processing parameter can be determined by the two model type parameters of the above model name and the above model path. It can be understood that when there is only one image processing model in the parsed model path, the reference image processing model can be directly determined according to the model type parameter of the model path.

[0053] In addition, the parsing result of the parsing operation also includes model description parameters, and the model description parameters include at least one of the following: input parameter identification parameter, output parameter identification parameter, number of convolution layers, number of convolution kernels, and convolution kernel size. By calling the constructor of the meta-algorithm template class and passing the model description parameters to the original algorithm template class, at least one reference image processing model is initialized to at least one target image processing model.

[0054] Further, when the target image processing model is determined, the above step S206 is executed to perform object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result; the specific process includes:

[0055] S206-1, when the parsing result carries a key tag parameter, determining the target object type according to the key tag parameter;

[0056] S206-2: When the recognition results of object recognition performed on the target image by at least one target image processing model include an object recognition result that matches the target object type, determine at least one model recognition result based on the object recognition result that matches the target object type.

[0057] It should be noted that the above key tag parameters are used to indicate the target object type of the recognition result. The following is an example of determining the model recognition result in the above steps S206-1 to S206-2.

[0058] Specifically, when the analysis result contains key label parameters, if the target image processing model detects that there are pedestrians in the current field of view, "person" is filled in the key label parameters. When "person" exists in the key label parameters, it is considered that there are pedestrians in the current field of view.

[0059] It can be understood that the above model recognition result includes the direct output image of the model and the detection information in the direct output image.

[0060] After the model recognition result is obtained through the above step S206, step S208 is executed to perform a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result. Specifically, the above step S208 specifically includes the following steps:

[0061] S208-1, obtaining an operation function indicated by the second image processing parameter;

[0062] S208-2: Perform a reference operation on at least one target operation according to the operation logic indicated by the operation function to obtain a target image processing result.

[0063] It can be understood that the above-mentioned second image processing parameter is used to indicate an operation function, and the above-mentioned operation function is used to further post-process the model recognition result obtained through step S206. For example, the target image processing model is responsible for vehicle detection. Through the above-mentioned step S206, vehicle detection information obtained by at least one target image processing model performing object recognition on the target image can be obtained. The vehicle detection information can be presented in the following form.

[0064] For example, the location information of the vehicle is marked in the form of a box in the detection image output by the target image processing model, and the coordinates of the four points of the box and the coordinates of the center point, as well as the vehicle confidence level, are given. Then, step S208-1 is executed to obtain the operation function indicated by the second image processing parameter. It should be noted that the above operation function can be further post-processing of the detection image output by the model, or it can be post-processing of the information of the detection object in the detection image output by the model, which is not specifically limited here.

[0065] As an optional implementation, in the case where the operation function is used to further post-process the detection image output by the model, the step S208-2 performs a reference operation on at least one target operation according to the operation logic indicated by the operation function to obtain a target image processing result, which specifically includes at least one of the following:

[0066] S20821-1, when the model recognition result is used to indicate the recognized target road object, determining at least one target result image according to a positional relationship between the target road object and the target image area;

[0067] S20821-2, performing image rendering operation on the model inference image to obtain a corresponding result image.

[0068] It can be understood that the above steps S20821-1 to S20821-2 are the process of further post-processing the detection image directly output by the model (model inference image) to obtain the above target result image, wherein the above step S20821-1 can be applied to determine whether the vehicle is in a no-parking area. Specifically, assuming that the recognition result of the model indicates that the vehicle object is recognized, the detection image directly output by the model is processed according to the positional relationship between the vehicle object and the preset no-parking area (the above target image area).

[0069] For example, when the vehicle object is completely in the no-parking area, a red detection frame is used to cover the detection frame of the vehicle in the detection image output by the model, and the image area outside the detection frame is blurred to highlight the illegally parked vehicle. For another example, when the vehicle object is partially in the no-parking area, a yellow detection frame is used to cover the detection frame of the vehicle in the detection image output by the model. When the vehicle object is not in the no-parking area at all, a green detection frame is used to cover the detection frame of the vehicle in the detection image output by the model.

[0070] It is worth noting that in the above step S20821-2, the image rendering operation can be to enhance the image contrast between the detection object and the background. For example, when the detection object is a pedestrian, when the above model inference image contains pedestrians, the detected pedestrian object is transformed in terms of color value and brightness to distinguish it from the background, and the rendered image is used as the model recognition result. Similarly, the background can also be transformed in terms of color value and brightness, or the pedestrian and the background can be transformed in terms of color value and brightness at the same time. No specific limitation is made here.

[0071] Through the above-mentioned implementation of the present application, the output image of the detection model can be processed by the operation function, including but not limited to determining at least one target result image according to the positional relationship between the target road object and the target image area when the model recognition result is used to indicate the recognized target road object; and / or performing image rendering operation on the model inference image to obtain the corresponding target result image. Thus, the user can more efficiently obtain important information from the output image.

[0072] As another optional implementation, in the case where the operation function is used to post-process the information of the detection object in the detection image output by the model, the step S208-2 performs a reference operation on at least one target operation according to the operation logic indicated by the operation function to obtain a target image processing result, and specifically includes at least one of the following:

[0073] S20822-1, after obtaining a first model recognition result for a first image region indicating where a first road object is located and a second model recognition result for a second image region indicating where a second road object is located, determining an object position relationship between the first road object and the second road object according to an image position relationship between the first image region and the second image region;

[0074] S20822-2, when a plurality of first model recognition results corresponding to a plurality of adjacent frame images and indicating a first image area where a first road object is located are obtained, and a plurality of second model recognition results corresponding to a plurality of adjacent frame images and indicating a second image area where a second road object is located are obtained, the relative motion relationship between the first road object and the second road object is determined according to the image position relationship between the first image area and the second image area in the plurality of adjacent frame images.

[0075] It should be noted that the above steps S208-1 to S208-2 are the above operation logic indicated by the second image processing parameter. It is worth noting that if the recognition results of at least two recognition objects are obtained through the above step S206, the above step S20822-1 can be executed. After obtaining the first model recognition result for indicating the first image area where the first road object is located and the second model recognition result for indicating the second image area where the second road object is located, the object position relationship between the first road object and the second road object is determined according to the image position relationship between the first image area and the second image area. Specifically, assuming that the vehicle is the first road object and the pedestrian is the second road object, the position relationship between the vehicle and the pedestrian is determined according to the image position relationship between the first image area where the vehicle is located and the second image area where the pedestrian is located. The determination method can be the distance between the coordinates of the center points of the image areas, or whether the image areas are crossed.

[0076] For example, in the case where the above-mentioned image position relationship is the distance between the coordinates of the center points of the image area, the distance between the midline points of the image area is determined as the distance between the pedestrian and the vehicle. For another example, in the case where the above-mentioned image position relationship is whether an intersection occurs, if there is an intersection area between the first image area where the vehicle is located and the second image area where the pedestrian is located, it is determined that the pedestrian and the vehicle may be in contact. If it is determined based on multiple adjacent time frame images that the current vehicle is in motion, it can be further confirmed that the pedestrian is currently in a dangerous state. It should be noted that the above two situations are not independent embodiments. The position relationship between the first road object and the second road object can be confirmed based on the two determination relationships of the distance between the coordinates of the center points of the image areas and whether the image areas intersect. No specific limitation is made here.

[0077] In addition, for the above-mentioned step S20822-2, it can be understood that the above-mentioned step S208-2 is to determine the relative motion relationship between the first road object and the second road object through the historical motion trajectory between the first image area and the second image area in multiple adjacent frame images. For example, assuming that the vehicle is the first road object and the pedestrian is the second road object, the positional relationship between the vehicle and the pedestrian is determined according to the image position relationship between the first image area where the vehicle is located and the second image area where the pedestrian is located in multiple adjacent frames. Specifically, assuming that, in the past 8 adjacent time frames, the center point distance between the first image area where the vehicle is located and the second image area where the pedestrian is located is less than a preset threshold, and the distance gradually decreases in the 8 adjacent time frames, it is determined that there is a relative movement between the vehicle and the pedestrian approaching each other.

[0078] Through the above optional implementation, the detection information of the detection object in the detection image output by the model can be further post-processed through the user-defined operation function, including but not limited to determining the object position relationship between the first road object and the second road object according to the image position relationship between the first image area and the second image area when the first model recognition result for indicating the first image area where the first road object is located and the second model recognition result for indicating the second image area where the second road object is located are obtained; determining the relative motion relationship between the first road object and the second road object according to the image position relationship between the first image area and the second image area in the multiple adjacent frame images when multiple first model recognition results corresponding to multiple adjacent frame images for indicating the first road object are located and multiple second model recognition results corresponding to multiple adjacent frame images for indicating the second road object are located are obtained; thereby directly outputting higher-order detection information, such as the relative distance between the detection objects, whether there is relative motion, etc., without the need for manual calculation, thereby improving the efficiency of model detection.

[0079] Through the above-mentioned implementation mode of the present application, an image processing instruction carrying a first image processing parameter and a second image processing parameter is first received; then, based on the first image processing parameter, at least one target image processing model indicated by the image processing instruction is determined from multiple candidate image processing models; then, object recognition is performed on the target image according to the at least one target image processing model to obtain at least one model recognition result; finally, according to the operation logic indicated by the second image processing parameter, a target operation is performed on the at least one model recognition result to obtain a target image processing result. Therefore, in the process of using the image processing model for image processing, when the image processing model needs to be replaced, there is no need to re-develop the model, thereby solving the technical problem of low flexibility in model replacement existing in the related art.

[0080] In an optional implementation, before performing object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result, the method further includes:

[0081] S1, determine the image processing function carried in the analysis result;

[0082] S2, performing image format conversion on the target image according to the image processing function to obtain a first image in a target format, wherein the target format is an image format that matches the target image processing model;

[0083] S3, normalizing the pixel values ​​of multiple pixels in the first image to obtain a second image to be input into the target image processing model.

[0084] It can be understood that the above-mentioned image processing function is an image conversion function, which is used to convert the format of the acquired image into an image format that can be used for feature extraction by the image processing model.

[0085] Specifically, first execute the above step S1 to determine the image processing function carried in the analysis result, and then execute step S2 to convert the target image into an image format through the above image processing function. For example, when the target image processing model is yolov8, the memory format of the target image is converted from the format of [R, G, B, R, G, B…] to the format of [R, R, R…G, G, G…B, B, B…], and then execute step S3. Specifically, the pixel values ​​of multiple pixels in the first image in the format of [R, R, R…G, G, G…B, B, B…] are normalized to obtain the second image to be input into the yolov8 model.

[0086] Through the above implementation of the present application, the image processing function carried in the analysis result can be first determined; then the image format of the target image is converted according to the image processing function to obtain a first image of the target format, wherein the target format is an image format that matches the target image processing model; finally, the pixel values ​​of multiple pixels in the first image are normalized to obtain a second image to be input into the target image processing model. Thus, the image to be detected can be received by the image processing model, and then in the process of switching the model, it is only necessary to call the image processing function corresponding to the switched model to enable the image to be detected to be received by the new image processing model.

[0087] In an optional implementation, after performing a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result, the method further includes:

[0088] S1, according to the running function carried by the image processing instruction, obtain the next image object as the target image;

[0089] S2, performing image processing on the target image according to the first image processing parameter and the second image processing parameter to obtain a target image processing result;

[0090] S3, when the current processing result satisfies the loop stopping condition, outputting the target image processing results corresponding to each of the multiple image objects.

[0091] It can be understood that the above steps S1 to S3 are loop execution of the above steps S202 to S208. Specifically, the above step S1 is first executed, and the next image object is obtained as the target image according to the operation function carried by the image processing instruction, and then the above step S2 is executed. The specific execution process of step S2 refers to the above steps S202 to S208, which will not be repeated here. Then, when the current processing result meets the loop stop condition, the target image processing results corresponding to each of the multiple image objects are output.

[0092] It can be understood that the above-mentioned loop stop conditions can be user-defined stop conditions, such as stopping the loop after a certain number of loops, stopping the loop after running for a period of time, stopping the loop at a certain time point, stopping the loop when the data received by an external sensor (such as a temperature sensor) exceeds a preset threshold, or stopping the loop when the energy consumption or load of the current system exceeds a threshold. No specific limitation is made here.

[0093] Figure 3 Schematic diagram of some embodiments of an image processing method of the present application, such as Figure 3 As shown, the system framework 312 is an abstract system framework for specifically implementing an image processing method of the present application. The system framework 312 can be roughly divided into three layers. The bottom layer is the model algorithm layer 306, which includes abstract classes of multiple different image reasoning models, such as abstract classes of Yolo5, Yolo8 and SSD models, and there are differences between different model algorithm classes; the middle layer is the model decoration layer 304, which is specifically a meta-algorithm template class. Through this layer, the differences between different models can be hidden based on the characteristics of the template, and the working layer 302 is the working process, which is responsible for executing step S302, and the algorithm loop reads the image for processing.

[0094] It should be noted that Figure 3 The abstract classes corresponding to the models in the model algorithm layer 306 are defined before the reasoning program is run, and when defining the abstract classes, the configuration files corresponding to the models also need to be defined. Specifically, before the image processing instruction carrying the first image processing parameter and the second image processing parameter is received, the following is also included:

[0095] S901, creating a target abstract class corresponding to at least one candidate image processing model, wherein the target abstract class is used to load the corresponding candidate image processing model in a target application;

[0096] S902: Create a target configuration file that matches at least one candidate image processing model, wherein the target configuration file is used to describe information of the candidate image processing model.

[0097] It should be noted that the model framework of the above-mentioned candidate image processing model is a model framework that can be run on the inference device, such as onnx, tensorrt, etc., which is not specifically limited here. If the model framework of the image processing model is a framework such as pytorch that cannot be used for inference on the inference device, it should first be converted into a model framework such as onnx, tensorrt, etc. that can be run directly on the inference device.

[0098] Furthermore, the above step S901, creating a target abstract class corresponding to at least one candidate image processing model, includes:

[0099] S901-1, according to the model structure of the candidate image processing model, create a target model loading function of a target abstract class matching the candidate image processing model, wherein the target model loading function is used to parse the model parameters of the candidate image processing model, and the model parameters include the input parameters and output parameters of the candidate image processing model, the number of convolution layers of the feature extraction network of the candidate image processing model, the number of convolution kernels of the feature extraction network, and the size of each convolution kernel of the feature extraction network;

[0100] S901-2, creating a label loading function of a target abstract class matching the candidate image processing model according to the target training label of the candidate image processing model, wherein the target training label is a training set label of the candidate image processing model in the model training phase, and the label loading function is used to convert the target training label into a dictionary type, and the dictionary type is used to describe the target training label with textual meaning;

[0101] S901-3, based on the image formats supported by the candidate image processing model, create an image conversion function of a target abstract class that matches the candidate image processing model, wherein the image conversion function is used to convert the image format of the image to be inferred into an image format supported by the candidate image processing model, and perform a pixel value normalization operation on all pixel points in the image to be inferred after the format conversion, to obtain a reference image to be inferred;

[0102] S901-4, creating an inference function of a target abstract class matching the candidate image processing model, wherein the inference function is used to perform an inference operation on a reference image to be inferred, and output current inference information according to a result of the inference operation, and the current inference information is used to describe position information of the inspected object in the image to be inferred;

[0103] S901-5, create an inference post-processing function of the target abstract class that matches the candidate image processing model, wherein the inference post-processing function is used to perform a data parsing operation on the current inference information, and obtain the coordinate value of the inspected object in the image to be inferred based on the result of the data parsing operation.

[0104] It should be noted that the above steps S901-1 to S901-5 do not constitute an execution sequence relationship, and the above steps S901-1 to S901-5 can be executed in any order without specific limitation. The following takes the creation of the target abstract class of the YoloV5 model as an example to specifically explain the above steps S901-1 to S901-5.

[0105] Define the target model loading function loadModel of the YoloV5 model file in the target abstract class corresponding to the YoloV5 model. For the YoloV5 model, this function will parse the model architecture of the YoloV5 model from the model file, including the name of the input parameter, the name of the output parameter, the number of convolution layers of the backbone model, and the number and size of the convolution kernels.

[0106] Define the label loading function loadLabelsList of the YoloV5 model label file in the target abstract class corresponding to the YoloV5 model. For the label file of the YoloV5 model trained on the user dataset, this function will load the content of the label file as a dictionary data type in the program, such as {0: "bus", 1: "person", 2: "vase"}. The dictionary is used to convert the digital labels inferred by the model into character labels with the meaning of this article.

[0107] Define the image conversion function prepareImage in the target abstract class corresponding to the YoloV5 model. For the YoloV5 model, this function will convert the image file into the image format supported by the YoloV5 model, mainly including the conversion of the image memory format from [R,G,B,R,G,B…] to [R,R,R…G,G,G…B,B,B…], and then normalize all pixel values ​​in the image, that is, convert the pixel value range from [0,255] to [0,1].

[0108] Define the inference function doInference in the target abstract class corresponding to the YoloV5 model. For the YoloV5 model, this function receives the image output by the image conversion function prepareImage, and then performs model inference to extract structured information from the image, such as the location information of pedestrians and vehicles.

[0109] The post-inference processing function postProcess is defined in the target abstract class corresponding to the YoloV5 model. For the YoloV5 model, this function is used to parse the position information output by the inference function doInference into the actual coordinate value on the original image.

[0110] Optionally, the target configuration file includes a plurality of reference fields, and the step S902 of creating a target configuration file that matches at least one candidate image processing model includes the following steps:

[0111] The target profile includes a plurality of reference fields, and a target profile is created that matches each of the at least one candidate image processing model, including:

[0112] S902-1, creating a name reference field, wherein the name reference field is used to describe the model name of the candidate image processing model;

[0113] S902-2, creating a type reference field, wherein the type reference field is used to describe the model type of the candidate image processing model;

[0114] S902-3, creating a label path reference field, where the label path reference field is used to describe the file path of the label file of the candidate image processing model on the current inference device;

[0115] S902-4, creating a model path reference field, wherein the model path reference field is used to describe the file path of the candidate image processing model on the current inference device;

[0116] S902-5, creating a target detection field, where the target detection field is used to indicate an object type of at least one detected object from a plurality of tag types.

[0117] It should be noted that the above steps S902-1 to S902-5 also do not constitute an execution sequence relationship, and the above steps S902-1 to S902-5 can be executed in any order, without specific limitation here. The following still takes the creation of the target abstract class of the YoloV5 model as an example to specifically explain the above steps S901-1 to S901-5.

[0118] Create a name reference field name, which should be the actual name of the model in the system, such as YoloV5, YoloV5.2.2, YoloV5ForPerson, and create a type reference field type, which should be the type of the model, that is, YoloV5. It can be understood that if the target configuration file of YoloV8 is created, this field should be YoloV8.

[0119] Create a label path reference field labelspath, which is the path of the label file, and create a model path reference field modelpath, which is the storage path of the YoloV5 model file. Create a target detection field keywords, which is the label name of the inferred target that the user's own algorithm logic is concerned about. For example, when using a visual model to infer whether there are pedestrians in the current field of view, "person" should be filled in the keyword. When the target label inferred by the model contains "person", it is considered that there are pedestrians in the current field of view.

[0120] It can be understood that, through the above steps S901 to S902, multiple candidate image processing models can be created, each corresponding to a target abstract class and a target configuration file.

[0121] Furthermore, based on the template feature of the C++ programming language, a meta-algorithm template class is defined. This class takes the above target abstract class as a template parameter, ignores the abstraction of the model, and unifies different models. Specifically:

[0122] First, define the meta-algorithm template class MetaAlgo. This class takes the model template as a parameter and determines its own template type at runtime. Define the template member parameter m_algo_ptr in this class. This member serves as a handle for class objects to access member functions and member variables of the meta-algorithm template class outside the class.

[0123] Afterwards, in the constructor MetaAlgo(Config&conf) of the meta-algorithm template class, the meta-algorithm template is initialized and m_algo_ptr is assigned according to the content of the parameter conf. The content of the parameter conf is the content of a candidate image processing model in the above target configuration file.

[0124] At the same time, the function void registerProcessor(function t_func) is defined in the meta-algorithm template class. The input parameter t_func of this function is a user-defined function. In the user-defined function, the user needs to implement his own target processing logic, such as judging whether the detected vehicle target is in a certain field of view. At the same time, there is a member variable m_processor in the meta-algorithm template class MetaAlgo, which will be assigned t_func, that is, m_processor=t_func, thereby realizing the transfer of user-defined logic from outside MetaAlgo to inside MetaAlgo.

[0125] In addition, it is necessary to define an entry class for parsing the above target configuration file and initializing the above meta-algorithm template class. Specifically, define the configuration file parsing function parseConfig, in which each field of the created target configuration file is read, and each field of the corresponding target configuration file is saved in a custom data structure. For example, the target configuration files of the YoloV5 and YoloV8 models are read, and each field of the target configuration files corresponding to the YoloV5 and YoloV8 models is saved in a custom data structure.

[0126] Define the user post-processing function userPostProcess in the entry class, in which the user adds custom logic operations. For example, determine whether the detected vehicle target is in the prohibited parking area, etc. Define the initialization function initAlgo in the entry class, in which the meta-algorithm template class is initialized according to the algorithm information stored in the configuration information data structure filled by the configuration file parsing function parseConfig, and assign the userPostProcess function to the m_processor member variable of the meta-algorithm template class to realize the transfer of user logic to the meta-algorithm template class; define the running function run in the entry class, in which the main process is started, and new images are read into the program in a continuous loop, and the algorithm processes the image in a loop and outputs the result.

[0127] The relationship between the above target abstract class, meta-algorithm template class, and entry class is as follows Figure 4 As shown, Figure 4 In the example, there are three target abstract classes, namely model abstract class 402, model abstract class 404, and model abstract class 406. These three target abstract classes may be target abstract classes corresponding to different candidate image processing models. For example, they are target abstract classes corresponding to YoloV5 model, YoloV8 model, and SSD model. It should be noted that Figure 4The three target abstract classes are only used as examples. In practice, there may be more or fewer target abstract classes, which are not specifically limited here.

[0128] Figure 4 In the example, model abstract class 402, model abstract class 404, and model abstract class 406 are used as templates for template class 412, and entry class 422 is used to pass user-defined functions to template class 412 and initialize template class 412. Figure 5 A completed image processing process is described.

[0129] First, execute steps S501-1 to S501-3 to load three candidate image processing models (for example, YoloV5, YoloV8, and SSD). Similarly, the candidate image processing models here are only for illustration purposes. The number of candidate image processing models in practice may be more or less, and no specific limitation is made here.

[0130] S502, initialization of the meta-algorithm template class, specifically, calling the initialization function initAlgo in the above entry class to initialize the meta-algorithm template class.

[0131] Step 503 - 1 to step S503 - 3, specifically, the meta-algorithm template class is configured using the target abstract classes corresponding to the three image processing models as templates.

[0132] S504, parse the configuration file and read the specific parameters of all models. This step is to parse the target configuration files corresponding to the three candidate image processing models and obtain the specific parameters corresponding to each of the three candidate image processing models, such as the name of the input parameter, the name of the output parameter, the number of convolution layers of the backbone model, the number and size of the convolution kernels, etc., which are not specifically limited here.

[0133] S506, instantiate all types of meta-algorithm template classes according to the configuration information. Specifically, meta-algorithm template classes of three target abstract class types, YoloV5, YoloV8, and SSD, can be instantiated here.

[0134] S508, the instantiated meta-algorithm template class calls the registerProcessor function to introduce the external user-defined functional logic. Specifically, the registerProcessor function is called to use the user-defined post-processing function userPostProcess in the above-mentioned entry class as the input parameter t_func of the registerProcessor function of the meta-algorithm template class, and the member variable m_processor in the meta-algorithm template class is assigned to t_func, that is, m_processor=t_func. In this way, the user-defined logic in the entry class is transferred from outside the template class to inside the template class. When the user needs to change the processing logic of the custom function, it is only necessary to adjust the entry class without adjusting the template class, which improves the efficiency of the process of changing the post-processing function.

[0135] S510, model inference and detection. Use the loaded inference model to detect the image to be inferred.

[0136] S512, use m_processor to perform further functional post-processing on the detected target. This step can be specifically referred to S208-2, and will not be described in detail here.

[0137] S514, result sending: The result obtained in step S512 is sent to the result receiving device.

[0138] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0139] According to another aspect of the embodiments of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Figure 6 As shown, the device comprises:

[0140] According to another aspect of the embodiment of the present application, there is also provided an image processing device, including:

[0141] A receiving unit 602, configured to receive an image processing instruction carrying a first image processing parameter and a second image processing parameter;

[0142] A determination unit 604 is used to determine at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models according to the first image processing parameter;

[0143] The recognition unit 606 is used to perform object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result;

[0144] The execution unit 608 is used to perform a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

[0145] Optionally, the execution unit 608 includes: an acquisition module, used to acquire an operation function indicated by the second image processing parameter; and a reference operation module, used to perform a reference operation on at least one target operation according to the operation logic indicated by the operation function to obtain a target image processing result.

[0146] Optionally, the above-mentioned determination unit 604 includes: a template parsing module, used to obtain a target configuration file indicated by the first image processing parameter, and parse the meta-algorithm template class; a model determination module, used to determine at least one reference image processing model from multiple candidate image processing models according to at least one model type parameter in the model description parameter set carried in the parsing result; an initialization module, used to initialize at least one reference image processing model to at least one target image processing model according to at least one model description parameter in the model description parameter set, wherein the at least one model description parameter includes at least one of the following: an input parameter identification parameter, an output parameter identification parameter, the number of convolution layers, the number of convolution kernels, and the convolution kernel size.

[0147] Optionally, the above-mentioned image processing device also includes: a normalization unit, which is used to: perform object recognition on the target image according to at least one target image processing model, and before obtaining at least one model recognition result, it also includes: determining the image processing function carried in the analysis result; performing image format conversion on the target image according to the image processing function to obtain a first image in a target format, wherein the target format is an image format that matches the target image processing model; and normalizing the pixel values ​​of multiple pixels in the first image to obtain a second image to be input into the target image processing model.

[0148] Optionally, the above-mentioned identification unit 606 also includes: a type determination module, which is used to determine the target object type according to the key label parameters when the parsing results carry the key label parameters; a result determination module, which is used to determine at least one model recognition result according to the object recognition result that matches the target object type when the recognition result of object recognition performed by at least one target image processing model on the target image includes an object recognition result that matches the target object type.

[0149] Optionally, the above-mentioned execution unit 608 is also used to, when the model recognition result is used to indicate the identified target road object, determine at least one target result image according to the positional relationship between the target road object and the target image area; and perform image rendering operation on the model inference image to obtain the corresponding target result image.

[0150] Optionally, the execution unit 608 is further configured to, upon obtaining a first model recognition result for indicating a first image region where the first road object is located and a second model recognition result for indicating a second image region where the second road object is located, determine the object position relationship between the first road object and the second road object according to the image position relationship between the first image region and the second image region; and upon obtaining a plurality of first model recognition results for indicating a first image region where the first road object is located corresponding to a plurality of adjacent frame images and a plurality of second model recognition results for indicating a second image region where the second road object is located corresponding to a plurality of adjacent frame images, determine the relative motion relationship between the first road object and the second road object according to the image position relationship between the first image region and the second image region in the plurality of adjacent frame images.

[0151] Optionally, the above-mentioned image processing device also includes a loop unit, which is used to obtain the next image object as the target image according to the running function carried by the image processing instruction; perform image processing on the target image according to the first image processing parameter and the second image processing parameter to obtain the target image processing result; when the current processing result meets the loop stop condition, output the target image processing results corresponding to each of the multiple image objects.

[0152] Optionally, the above-mentioned image processing device also includes a creation unit, which is used to create a target abstract class corresponding to at least one candidate image processing model, wherein the target abstract class is used to load the corresponding candidate image processing model in the target application; and create a target configuration file that matches each of the at least one candidate image processing models, wherein the target configuration file is used to describe the information of the candidate image processing model.

[0153] Optionally, the creation unit is further used to: create a target model loading function of a target abstract class that matches the candidate image processing model according to the model structure of the candidate image processing model, wherein the target model loading function is used to parse the model parameters of the candidate image processing model, and the model parameters include the input parameters, output parameters, the number of convolution layers of the feature extraction network of the candidate image processing model, the number of convolution kernels of the feature extraction network, and the size of each convolution kernel of the feature extraction network; create a label loading function of a target abstract class that matches the candidate image processing model according to the target training label of the candidate image processing model, wherein the target training label is the training set label of the candidate image processing model in the model training stage, and the label loading function is used to convert the target training label into a dictionary type, and the dictionary type is used to describe the target training label with textual meaning; Support image formats, create an image conversion function of the target abstract class that matches the candidate image processing model, wherein the image conversion function is used to convert the image format of the image to be inferred into the image format supported by the candidate image processing model, and perform pixel value normalization operations on all pixels in the image to be inferred after the format conversion to obtain a reference image to be inferred; create an inference function of the target abstract class that matches the candidate image processing model, wherein the inference function is used to perform inference operations on the reference image to be inferred, and output current inference information based on the result of the inference operation, and the current inference information is used to describe the position information of the inspected object in the image to be inferred; create an inference post-processing function of the target abstract class that matches the candidate image processing model, wherein the inference post-processing function is used to perform data parsing operations on the current inference information, and obtain the coordinate value of the inspected object in the image to be inferred based on the result of the data parsing operation.

[0154] Optionally, the above-mentioned creation unit is also used to: create a name reference field, wherein the name reference field is used to describe the model name of the candidate image processing model; create a type reference field, wherein the type reference field is used to describe the model type of the candidate image processing model; create a label path reference field, wherein the label path reference field is used to describe the file path of the label file of the candidate image processing model on the current inference device; create a model path reference field, wherein the model path reference field is used to describe the file path of the candidate image processing model on the current inference device; create a target detection field, wherein the target detection field is used to indicate the object type of at least one inspected object from multiple label types.

[0155] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned method for processing data in a memory is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a mobile phone or a computer as an example. Figure 7As shown, the electronic device includes a memory 702 and a processor 704. The memory 702 stores a computer program, and the processor 704 is configured to execute the steps in any of the above method embodiments through the computer program.

[0156] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0157] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0158] S1, receiving an image processing instruction carrying a first image processing parameter and a second image processing parameter;

[0159] S2, determining at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models according to the first image processing parameter;

[0160] S3, performing object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result;

[0161] S4, performing a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

[0162] Alternatively, a person skilled in the art may understand that: Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 6 Different configurations are shown.

[0163] Among them, the memory 702 can be used to store software programs and modules, such as program instructions / modules corresponding to the image processing method and device in the embodiments of the present application. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, realizing the above-mentioned image processing method. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 702 may further include a memory remotely located relative to the processor 704, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 702 can be specifically, but not limited to, used to store information such as page elements and page styles. As an example, such as Figure 7 As shown, the memory 702 may include but is not limited to the receiving unit 602, the determining unit 604, the identifying unit 606 and the executing unit 08 in the image processing device. In addition, it may also include but is not limited to other module units in the image processing device, which will not be repeated in this example.

[0164] Optionally, the transmission device 706 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 706 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0165] In addition, the electronic device further includes: a display 708 for displaying the target page; and a connection bus 710 for connecting various module components in the electronic device.

[0166] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a point-to-point network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the point-to-point network.

[0167] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations;

[0168] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0169] S1, receiving an image processing instruction carrying a first image processing parameter and a second image processing parameter;

[0170] S2, determining at least one target image processing model indicated by the image processing instruction from a plurality of candidate image processing models according to the first image processing parameter;

[0171] S3, performing object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result;

[0172] S4, performing a target operation on at least one model recognition result according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

[0173] Optionally, in the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0174] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0176] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0181] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0182] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0183] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0184] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0185] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0186] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0187] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0188] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0189] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0190] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0191] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: receiving an image processing instruction carrying a first image processing parameter and a second image processing parameter; Obtaining a target configuration file indicated by the first image processing parameter, and parsing a meta-algorithm template class; Determine at least one reference image processing model from a plurality of candidate image processing models according to at least one model type parameter in a model description parameter set carried in the parsing result; Initializing at least one of the reference image processing models as at least one target image processing model according to at least one model description parameter in the model description parameter set, wherein at least one of the model description parameters includes at least one of the following: an input parameter identification parameter, an output parameter identification parameter, the number of convolution layers, the number of convolution kernels, and the size of the convolution kernel; Performing object recognition on the target image according to at least one of the target image processing models to obtain at least one model recognition result; According to the operation logic indicated by the second image processing parameter, a target operation is performed on at least one of the model recognition results to obtain a target image processing result.

2. The method according to claim 1, characterized in that The step of performing a target operation on at least one of the model recognition results according to the operation logic indicated by the second image processing parameter to obtain a target image processing result includes: Obtaining an operation function indicated by the second image processing parameter; A reference operation is performed on at least one of the target operations according to the operation logic indicated by the operation function to obtain a target image processing result.

3. The method according to claim 2, characterized in that Before performing object recognition on the target image according to at least one target image processing model to obtain at least one model recognition result, the method further includes: Determine the image processing function carried in the analysis result; Performing image format conversion on the target image according to the image processing function to obtain a first image in a target format, wherein the target format is an image format that matches the target image processing model; The pixel values ​​of multiple pixel points in the first image are normalized to obtain a second image to be input into the target image processing model.

4. The method according to claim 2, characterized in that: The performing object recognition on the target image according to at least one of the target image processing models to obtain at least one model recognition result includes: When the parsing result carries a key tag parameter, determining the target object type according to the key tag parameter; In the case where the recognition results of at least one of the target image processing models performing object recognition on the target image include an object recognition result that matches the target object type, at least one of the model recognition results is determined based on the object recognition result that matches the target object type.

5. The method according to claim 2, characterized in that: The performing a reference operation on at least one of the target operations according to the operation logic indicated by the operation function to obtain a target image processing result includes at least one of the following: In a case where the model recognition result is used to indicate the recognized target road object, determining at least one target result image according to a positional relationship between the target road object and a target image area; Perform image rendering operation on the model inference image to obtain the corresponding target result image.

6. The method according to any one of claims 1 to 5, characterized in that The performing a target operation on at least one of the model recognition results according to the operation logic indicated by the second image processing parameter to obtain a target image processing result further includes at least one of the following: determining an object position relationship between the first road object and the second road object according to an image position relationship between the first image area and the second image area, when a first model recognition result indicating a first image area where a first road object is located and a second model recognition result indicating a second image area where a second road object is located are obtained; When a plurality of first model recognition results corresponding to a plurality of adjacent frame images and indicating a first image area where a first road object is located are obtained, and a plurality of second model recognition results corresponding to a plurality of adjacent frame images and indicating a second image area where a second road object is located are obtained, the relative motion relationship between the first road object and the second road object is determined based on the image position relationship between the first image area and the second image area in the plurality of adjacent frame images.

7. The method according to claim 6, characterized in that After performing a target operation on at least one of the model recognition results according to the operation logic indicated by the second image processing parameter to obtain a target image processing result, the method further includes: According to the running function carried by the image processing instruction, obtaining the next image object as the target image; Performing image processing on the target image according to the first image processing parameter and the second image processing parameter to obtain a target image processing result; When the current processing result satisfies the loop stopping condition, the target image processing results corresponding to each of the multiple image objects are output.

8. The method according to claim 1, characterized in that: Before receiving the image processing instruction carrying the first image processing parameter and the second image processing parameter, the method further includes: Creating a target abstract class corresponding to at least one of the candidate image processing models, wherein the target abstract class is used to load the corresponding candidate image processing model in a target application; A target configuration file is created that matches at least one of the candidate image processing models, wherein the target configuration file is used to describe information of the candidate image processing model.

9. The method according to claim 8, characterized in that Creating a target abstract class corresponding to at least one candidate image processing model, wherein the target abstract class is used to load the corresponding candidate image processing model in a target application, comprising: According to the model structure of the candidate image processing model, a target model loading function of a target abstract class matching the candidate image processing model is created, wherein the target model loading function is used to parse the model parameters of the candidate image processing model, and the model parameters include the input parameters and output parameters of the candidate image processing model, the number of convolutional layers of the feature extraction network of the candidate image processing model, the number of convolutional kernels of the feature extraction network, and the size of each convolutional kernel of the feature extraction network; According to the target training label of the candidate image processing model, a label loading function of a target abstract class matching the candidate image processing model is created, wherein the target training label is a training set label of the candidate image processing model in the model training stage, and the label loading function is used to convert the target training label into a dictionary type, and the dictionary type is used to describe the target training label with textual meaning; According to the image formats supported by the candidate image processing model, an image conversion function of a target abstract class matching the candidate image processing model is created, wherein the image conversion function is used to convert the image format of the image to be inferred into the image format supported by the candidate image processing model, and perform a pixel value normalization operation on all pixel points in the image to be inferred after the format conversion, so as to obtain a reference image to be inferred; Creating an inference function of a target abstract class matching the candidate image processing model, wherein the inference function is used to perform an inference operation on the reference image to be inferred, and output current inference information according to a result of the inference operation, wherein the current inference information is used to describe position information of an object to be inspected in the image to be inferred; Create an inference post-processing function of a target abstract class that matches the candidate image processing model, wherein the inference post-processing function is used to perform a data parsing operation on the current inference information and obtain the coordinate value of the inspected object in the image to be inferred based on the result of the data parsing operation.

10. The method according to claim 8, characterized in that The target configuration file includes a plurality of reference fields, and the step of creating a target configuration file that matches each of the at least one candidate image processing models includes: Creating a name reference field, wherein the name reference field is used to describe the model name of the candidate image processing model; Creating a type reference field, wherein the type reference field is used to describe the model type of the candidate image processing model; Creating a label path reference field, wherein the label path reference field is used to describe the file path of the label file of the candidate image processing model on the current inference device; Creating a model path reference field, wherein the model path reference field is used to describe the file path of the candidate image processing model on the current inference device; An object detection field is created, wherein the object detection field is used to indicate an object type of at least one detected object from a plurality of tag types.

11. An image processing device, characterized in that: include: A receiving unit, configured to receive an image processing instruction carrying a first image processing parameter and a second image processing parameter; A determination unit, configured to obtain a target configuration file indicated by the first image processing parameter, and parse the meta-algorithm template class; determine at least one reference image processing model from a plurality of candidate image processing models according to at least one model type parameter in a model description parameter set carried in a parsing result; initialize at least one of the reference image processing models to at least one target image processing model according to at least one model description parameter in the model description parameter set, wherein at least one of the model description parameters includes at least one of the following: an input parameter identification parameter, an output parameter identification parameter, the number of convolution layers, the number of convolution kernels, and the size of the convolution kernel; an identification unit, configured to perform object identification on a target image according to at least one of the target image processing models, and obtain at least one model identification result; An execution unit is used to perform a target operation on at least one of the model recognition results according to the operation logic indicated by the second image processing parameter to obtain a target image processing result.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 10 when executed by a processor.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

Images