Image processing model training method and image processing method

Abstract

Provided in the embodiments of the present specification are an image processing model training method and an image processing method. The image processing model training method comprises: acquiring a sample image and a sample image processing result corresponding to the sample image; inputting the sample image into an image processing model, so as to obtain a predicted image processing result which is output from the image processing model, wherein the predicted image processing result is generated on the basis of first angle-of-view image feature information and second angle-of-view image feature information, which first angle-of-view image feature information is generated on the basis of the sample image, and which second angle-of-view image feature information is generated on the basis of the first angle-of-view image feature information; calculating a model loss value on the basis of the predicted image processing result and the sample image processing result; and adjusting model parameters of the image processing model on the basis of the model loss value, and continuing to train the image processing model until a model training stop condition is met. In this way, the image processing model can be trained by means of making full use of images in a plurality of angles of view, thereby improving the accuracy of the image processing model during image processing.

Description

Image processing model training method and image processing method

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 1, 2023, with application number 202310961569.3 and application name “Image processing model training method and image processing method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of computer technology, and in particular to an image processing model training method. Background Art

[0003] With the rapid development of drone technology, drones have been widely used in research and industrial applications. Images and videos are important ways for drones to perceive their surroundings. Image semantic segmentation is a research hotspot in computer vision, with widespread applications in scenarios such as autonomous driving and intelligent robotics. Semantic segmentation of drone aerial images uses semantic segmentation techniques based on drone aerial imagery to enable drones to acquire intelligent perception capabilities of objects in a scene.

[0004] However, drones are often in motion during flight, causing the perspective of the images they capture to constantly change. This results in existing image processing methods for drones being less accurate than expected. Therefore, those skilled in the art need a more accurate image processing method for drones.

[0005] Summary of the Invention

[0006] In view of this, embodiments of this specification provide an image processing model training method, an image processing method, and a target object recognition method. One or more embodiments of this specification also relate to an image processing model training apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0007] According to a first aspect of an embodiment of this specification, there is provided an image processing model training method, comprising:

[0008] Obtaining a sample image and a sample image processing result corresponding to the sample image;

[0009] Inputting the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information;

[0010] Calculating a model loss value based on the predicted image processing result and the sample image processing result;

[0011] Adjust the model parameters of the image processing model according to the model loss value, and continue to train the image processing model until the model training stopping condition is reached.

[0012] According to a second aspect of the embodiments of this specification, there is provided an image processing method, including:

[0013] receiving an image processing request sent by a user, wherein the image processing request carries an image to be processed;

[0014] Inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model, wherein the image processing model is obtained by the image processing model training method described in the first aspect of the above embodiment;

[0015] The image processing result is sent to the user.

[0016] According to a third aspect of the embodiments of this specification, a method for identifying a target object is provided, including:

[0017] Obtain images to be processed collected by the drone;

[0018] Inputting the image to be processed into an image processing model to obtain a water surface area and a target object area in the image to be processed output by the image processing model, wherein the image processing model is obtained by the image processing model training method described in the first aspect of the above embodiment;

[0019] generating a shore area according to the water surface area;

[0020] The recognition result of the target object is determined according to the comparison result between the target object area and the shore area.

[0021] According to a fourth aspect of the embodiments of this specification, there is provided an image processing model training device, comprising:

[0022] An acquisition module, which acquires a sample image and a sample image processing result corresponding to the sample image;

[0023] a prediction module, inputting the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-view image feature information and second-view image feature information, wherein the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information;

[0024] A judgment module, which calculates a model loss value according to the predicted image processing result and the sample image processing result;

[0025] A stop module adjusts the model parameters of the image processing model according to the model loss value, and continues to train the image processing model until a model training stop condition is reached.

[0026] According to a fifth aspect of the embodiments of this specification, there is provided a computing device, including:

[0027] memory and processor;

[0028] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned image processing model training method or image processing method or target object recognition method are implemented.

[0029] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0030] According to the seventh aspect of the embodiments of this specification, a computer program is provided, wherein, when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0031] An image processing model training method provided by an embodiment of the present specification obtains a sample image and a sample image processing result corresponding to the sample image; inputs the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-perspective image feature information and second-perspective image feature information, the first-perspective image feature information is generated based on the sample image, and the second-perspective image feature information is generated based on the first-perspective image feature information; calculates a model loss value based on the predicted image processing result and the sample image processing result; adjusts the model parameters of the image processing model based on the model loss value, and continues to train the image processing model until a model training stop condition is reached.

[0032] According to the method provided in the embodiments of this specification, first-perspective image feature information is generated based on the sample image, and then second-perspective image feature information is obtained based on the first-perspective image feature information, and the image processing result is predicted based on the first-perspective image feature information and the second-perspective image feature information, and the first-perspective image feature information and the second-perspective image feature information are combined to perform image prediction. This can be understood as converting the sample image features that originally only had a single perspective into image features with multiple perspectives, and then combining the original single-perspective image features with the multiple-perspective image features and inputting them into the next layer of the model. This can enable the image processing model to fully utilize images from multiple perspectives for training, and thus can improve the accuracy of the image processing model trained using the image processing model training method when processing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is an architecture diagram of an image processing model training system provided by one embodiment of this specification;

[0034] FIG2 is a flow chart of an image processing model training method provided by one embodiment of this specification;

[0035] FIG3 is a structural diagram of an image processing model provided by one embodiment of this specification;

[0036] FIG4 is a flowchart of a process of a target object identification method provided by one embodiment of this specification;

[0037] FIG5 is a flowchart of a processing process of a semantic segmentation model training method provided by one embodiment of this specification;

[0038] FIG6 is a flowchart of an image processing method provided by one embodiment of this specification;

[0039] FIG7 is a schematic diagram of the structure of an image processing model training device provided by one embodiment of this specification;

[0040] FIG8 is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0041] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0042] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] First, the terms involved in one or more embodiments of this specification are explained.

[0046] ViT (Vision Transformer): An image processing model based on the Transformer architecture. Initially, the Transformer model was primarily applied to natural language processing tasks, but in recent years, researchers have found that it also performs well in computer vision tasks. The ViT model works by dividing an image into small patches, flattening these patches into one-dimensional vectors, and then feeding these vectors into the Transformer model for processing.

[0047] SETR (SEgmentation Transformer): A semantic segmentation model based on the Transformer architecture.

[0048] SegFormer: A Transformer-based semantic segmentation model that combines Transformer with convolutional neural networks to achieve more efficient and accurate image segmentation.

[0049] Morphological Expansion: Morphological dilation is a method used in image processing. Before dilation, the image is first binarized to obtain the target object. Morphological dilation expands the boundary points of the binary object, merging all background points that touch the object into the object, expanding the boundary outward. If two objects are close together, the object is connected. Different dilation distances produce different sizes of the dilated object boundary.

[0050] In this specification, an image processing model training method is provided. This specification also involves an image processing model training device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0051] Referring to FIG1 , FIG1 shows an architecture diagram of an image processing model training system provided by an embodiment of the present specification. The image processing model training system may include a client 100 and a server 200;

[0052] The client 100 is configured to send a sample image and a sample image processing result corresponding to the sample image to the server 200;

[0053] The server 200 is configured to input the sample image into an image processing model, obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-view image feature information and second-view image feature information, wherein the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information; calculate a model loss value based on the predicted image processing result and the sample image processing result; adjust model parameters of the image processing model based on the model loss value, and continue training the image processing model until a model training stop condition is reached; and send the model parameters of the image processing model to the client 100;

[0054] The client 100 is further configured to receive model parameters of the image processing model sent by the server 200 .

[0055] By applying the solution of the embodiments of this specification, first-perspective image feature information is generated based on the sample image, and then second-perspective image feature information is obtained based on the first-perspective image feature information, and the image processing result is predicted based on the first-perspective image feature information and the second-perspective image feature information, and the first-perspective image feature information and the second-perspective image feature information are combined for image prediction. This can be understood as converting the sample image features that originally had only a single perspective into image features with multiple perspectives, and then combining the original single-perspective image features with the multiple-perspective image features and inputting them into the next layer of the model. This can enable the image processing model to fully utilize images from multiple perspectives for training, and thus improve the accuracy of the image processing model trained using the image processing model training method when processing images.

[0056] The image processing system may include multiple clients 100 and a server 200. The clients 100 may be referred to as end-side devices, and the server 200 may be referred to as cloud-side devices. Multiple clients 100 may establish communication connections through the server 200. In image processing scenarios, the server 200 provides image processing services to multiple clients 100. Multiple clients 100 may function as either senders or receivers, communicating through the server 200.

[0057] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In the image processing scenario, users can publish data streams to the server 200 through the client 100. The server 200 generates image processing results based on the data stream and pushes the image processing results to other clients with which communication has been established.

[0058] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, or other processing before being released to the server 200.

[0059] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be based on the software development kit (SDK) of the corresponding service provided by the server 200, such as developed based on the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0060] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server that is integrated with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0061] It is worth noting that the image processing model training methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server to execute the image processing model training methods provided in the embodiments of this specification. In other embodiments, the image processing model training methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0062] Referring to FIG. 2 , FIG. 2 shows a flowchart of an image processing model training method provided by one embodiment of this specification, which specifically includes the following steps:

[0063] Step 202: Obtain a sample image and a sample image processing result corresponding to the sample image.

[0064] In actual applications, the image processing tasks sent by users can be received through the server or the client.

[0065] Specifically, the training method provided in this specification is supervised training, which includes sample images and sample processing results corresponding to the sample images. Specifically, the sample images are arbitrary image inputs, and the corresponding sample processing results are determined by a variety of image processing tasks, including semantic segmentation, key point detection, image generation, etc.

[0066] It should be noted that the sample images include any color images or grayscale images, video clips composed of multiple images, etc. The processing results of the sample images have different results depending on the classification of the task corresponding to the image processing. For example, the sample processing result corresponding to the semantic segmentation task is an image with the same size as the sample image, in which each pixel is marked with a category. Common categories include people, cars, roads, buildings, etc. The processing result of key point detection is usually a key point position annotation, which is used to specify the position of a specific key point in the image. Key points can be some specific points, such as facial key points (such as eyes, nose, mouth, etc.), human joints (such as shoulders, elbows, knees, etc.) or other key points. The annotation method can use pixel coordinates or normalized coordinates to represent the position of the key point.

[0067] In one embodiment provided in this specification, the sample image may be an image captured by a drone, and the corresponding image processing result may be an image carrying water surface markers and building markers. The required building sequence is detected by processing the water surface and judging the building.

[0068] By receiving a sample image, obtaining a predicted sample image processing result based on the obtained sample image, and comparing the predicted sample image processing result with the obtained sample image processing result, adjusting the model parameters according to the comparison result, the model training is completed.

[0069] Step 204: Input the sample image into the image processing model to obtain the predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on the first-perspective image feature information and the second-perspective image feature information, the first-perspective image feature information is generated based on the sample image, and the second-perspective image feature information is generated based on the first-perspective image feature information.

[0070] In practical applications, the sample image is input into the image processing model, and a predicted image processing result is generated based on the first-view image feature information and the second-view image feature information of the sample image.

[0071] Specifically, the first-view image feature information is a feature matrix derived from the sample image through the image processing model, while the second-view image feature is a feature matrix derived from the first-view image. Furthermore, the first-view image feature information can be understood as a feature matrix containing the basic characteristics of the sample image, used for subsequent processing by the image processing model. The second-view image feature information can be understood as a feature matrix after processing the first-view image feature information, used to enhance the image processing model's ability to process images from multiple different viewpoints.

[0072] In one embodiment provided herein, image processing tasks can be applied to semantic segmentation scenarios, where the locations of key points in an image that match the annotations provided by a technician can be determined. For example, in an application scenario where a drone is detecting buildings near a shoreline, the multi-view images collected by the drone can be used to detect image regions labeled "water" and "buildings," thereby confirming that the buildings in the image are near the shoreline. This helps technicians distinguish these buildings and facilitates subsequent management.

[0073] For example, in the application scenario of archaeological exploration, drones can take high-resolution surface images at the archaeological site, identify possible ruins, relics and geographical features through image classification technology, and thus assist archaeologists in exploration work.

[0074] In practical applications, the collected images can be any color images or grayscale images, etc.

[0075] By inputting the sample image into the image processing model, a predicted image processing result can be generated based on the first-perspective image feature information and the second-perspective image feature information of the sample image, thereby improving the accuracy of the image processing model when processing images from multiple different perspectives.

[0076] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the image processing model may include a basic module and a perspective generation module;

[0077] Furthermore, the sample image is input into an image processing model to obtain a predicted image processing result output by the image processing model, including:

[0078] Inputting the sample image into the basic module to obtain first-view image feature information;

[0079] Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information;

[0080] The first-view image feature information and the second-view image feature information are input into the basic module to obtain a predicted image processing result.

[0081] Referring to Figure 3, which illustrates a schematic diagram of the structure of an image processing model provided by one embodiment of this specification, the image processing model includes a basic module 302 and a perspective generation module 304. In practical applications, first-perspective image feature information is generated for a sample image in basic module 302. This first-perspective image feature information is then input into perspective generation module 304 to generate second-perspective image feature information. Both the first-perspective image feature information and the second-perspective image feature information are then input into basic module 302 to obtain a predicted image processing result.

[0082] Specifically, the basic module is any image processing model based on the Transformer architecture and including at least two encoder cores, such as ViT, SETR, SegFormer, etc., and this specification does not impose any restrictions here. The perspective generation module is a module that processes the first perspective feature information to obtain the second perspective information. Furthermore, the basic module can be understood as the basic architecture when using the above-mentioned image processing model to perform model training tasks and image processing tasks, and the perspective generation module can be understood as a module that improves the model's ability to obtain information from multiple perspective images during the training process of the image processing model.

[0083] By adding a perspective generation module during image processing model training, the model's ability to acquire information from multiple perspective images is improved, thereby improving the accuracy of image processing performed by the obtained image processing model.

[0084] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the basic module may include an encoder and a decoder;

[0085] Specifically, the sample image is input into the basic module to obtain first-view image feature information, including:

[0086] The sample image is input into the encoder to obtain first-view image feature information.

[0087] 3 , as shown in FIG3 , the basic module 302 includes an encoder 306 and a decoder 308. In practical applications, a sample image is input into the encoder 306 to obtain first-view image feature information.

[0088] Specifically, the encoder is used to process the sample image into the feature information corresponding to the sample image, and the decoder is used to process the image feature information obtained by the encoder into the information required by the task type. Furthermore, the encoder can be understood as extracting the features of the sample image and processing them according to the parameters of the model, and the decoder can be understood as processing the feature matrix processed by the encoder into the result required by the task type. For example, if the task type is a semantic segmentation task, the role of the decoder is to process the feature matrix processed by the encoder into an image with a specific target identifier. For example, if the task type is an image classification task, the role of the decoder is to process the feature matrix processed by the encoder into an image with a classification identifier.

[0089] By encoding the sample image, the computer can extract the features of the sample image, and by decoding the features, the computer can realize the actual processing of the sample image.

[0090] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the encoder may include an embedding layer and at least two encoding layers;

[0091] Specifically, the sample image is input into the encoder to obtain first-view image feature information, including:

[0092] Inputting the sample image into the embedding layer to obtain sample image encoding information;

[0093] Inputting the sample image coding information into an initial coding layer set to obtain initial coding image feature information, wherein the initial coding layer set includes an integer number of coding layers;

[0094] The initial coded image feature information is input into the first coding layer to obtain the first-view image feature information.

[0095] 3 , as shown in FIG3 , encoder 306 includes an embedding layer 310, an initial coding layer set 312, a first coding layer 314, a second coding layer 316, and a target coding layer set 318. In practical applications, the sample image undergoes an embedding operation in embedding layer 310 to obtain sample image coding information, which is then input into the initial coding layer set to obtain initial coding image feature information, and the initial coding image feature information is then input into first coding layer 314 to obtain first-view feature information.

[0096] Specifically, the embedding layer is an image processing layer that processes the sample image into sample image encoding information. The initial encoding layer set is a set of encoding layers containing an integer number of encoding layers. The first encoding layer is the encoding layer that obtains the first-view feature information and does not specifically refer to the encoding layer at a specific position. Furthermore, the embedding layer can be understood as performing matrix processing on the sample image to facilitate subsequent encoding operations. The encoding layer can be understood as performing encoding operations on the matrixed image after the embedding layer processing, outputting information with the sample image characteristics.

[0097] It should be noted that the embedding layer refers to the matrix operation on the sample image, which is related to the model used by the basic module. For example, if the model used by the basic module is the SegFormer model, then the embedding layer is a lightweight CNN encoder in the SegFormer model to extract the multi-scale features of the image and a word segmentation embedding unit to perform word embedding operations on the multi-scale features extracted by CNN. If the model used by the basic module is the ViT model, then the embedding layer is the word segmentation embedding unit in the ViT model to perform word embedding operations on the sample image.

[0098] In one embodiment provided in this specification, the basic module uses the SegFormer model and sets the output coding layer set to zero, so after the image processing receives the sample image, it first inputs the sample image into a simple CNN encoder to obtain the multi-scale features of the sample image, and then inputs the multi-scale features into the word segmentation embedding unit to obtain the sample image encoding information corresponding to the sample image, and then inputs the obtained sample image encoding information into the first encoder to obtain the first-view image feature information.

[0099] By performing matrix operations on the sample image, the image processing model can process the sample image more easily.

[0100] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective generation module may include a perspective extraction layer, a perspective management layer, and a perspective transformation layer;

[0101] Furthermore, inputting the first-view image feature information into the view generation module to obtain the second-view image feature information includes:

[0102] Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information;

[0103] Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information;

[0104] Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information;

[0105] The target perspective feature information is input into the perspective extraction layer to generate second perspective image feature information.

[0106] Referring to FIG. 3 , as shown in FIG. 3 , the perspective generation module 304 includes a perspective extraction layer 320, a perspective transformation layer 324, and a perspective management layer 322. In practical applications, first-perspective image information can be input into the perspective extraction layer 320 to generate initial perspective feature information, which is then input into the perspective management layer 322 to generate reference perspective feature information, which is then input into the perspective transformation layer 324 to generate target perspective feature information, and finally, the target perspective feature information is input into the perspective extraction layer to generate second-perspective image feature information.

[0107] Specifically, the perspective extraction layer can encode the first-perspective image feature information passed to the perspective generation module into initial perspective feature information through the encoding core, or it can decode the target perspective feature information generated in the perspective generation module through the decoding core and output it to the perspective generation module and pass it to the second encoding layer. The perspective management layer can receive the initial perspective feature information encoded by the perspective extraction layer and perform multi-dimensional processing on the received initial perspective feature information to obtain reference target feature information. The perspective transformation layer receives the perspective reference target feature information and obtains the target perspective feature information through dimensional transformation based on the dimensional information of the perspective management layer.

[0108] By performing perspective generation operations on sample images, the image processing model fully utilizes images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images.

[0109] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective extraction layer may include a perspective encoding unit and a perspective decoding unit;

[0110] Specifically, inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information includes:

[0111] Inputting the first viewing angle image information into the viewing angle encoding unit to generate initial viewing angle feature information;

[0112] Accordingly, the target perspective feature information is input into the perspective extraction layer to generate second perspective image feature information, including:

[0113] The target viewing angle feature information is input into the viewing angle decoding unit to generate second viewing angle image feature information.

[0114] 3 , as shown in FIG3 , the view extraction layer 320 includes a view encoding unit 326 and a view decoding unit 328. In practical applications, the first view image information can be input into the view encoding unit 326 to obtain initial view feature information for subsequent view generation processing, and the target view feature information after view generation can also be decoded by the decoding unit 328 to generate the second view image information.

[0115] Specifically, the encoding unit encodes the first-perspective image information, and the decoding unit decodes the target-perspective feature information. Furthermore, the encoding unit can be understood as extracting the core information from the first-perspective image information and compressing it to facilitate subsequent processing based on the central dimension. The decoding unit can be understood as restoring the target-perspective feature information generated based on the perspective to the second-perspective image feature information of the same length as the first-perspective image feature information.

[0116] It should be noted that the encoding unit and decoding unit are both the encoder cores in a Transformer model, and their purpose is to change the length of the received data while ensuring the integrity of the original information as much as possible.

[0117] The information received and sent by the view generation module is processed by the view extraction layer, which effectively improves the operating efficiency of the view generation module.

[0118] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective management layer may include at least one perspective dimension information;

[0119] Specifically, the initial viewing angle feature information is input into the viewing angle management layer to generate reference viewing angle feature information, including:

[0120] Reference viewing angle feature information is generated according to the initial viewing angle feature information and each viewing angle dimension information.

[0121] In practical applications, the view management layer receives initial view feature information and can generate reference view feature information based on information of each dimension.

[0122] Specifically, the dimensional information in the perspective management layer is initialized based on the initial input sample images at the beginning of training, and is adjusted according to the input sample images during the training process. The number of dimensional perspective information is an arbitrary positive integer. Furthermore, the role of the perspective management layer can be understood as managing the perspective dimension through the input sample images, so that the perspective dimension information can be more closely aligned with the actual application scenario, thereby improving the accuracy of perspective dimension processing. The number of dimensions can be understood as a preliminary classification of the sample images according to the dimensions, and the sample images are processed according to the classified dimensional information to fuse image features of different dimensions.

[0123] In one embodiment provided in this specification, the number of dimensions is set to 64. At the beginning of training, 64 initial perspective feature information based on the initial input perspective management layer are initialized to 64 perspective dimension parameters, the initial perspective feature information subsequently input is classified according to the above 64 perspective dimensions, the perspective dimension parameters are adjusted based on the classification of the perspective dimensions, and the parameters of the 64 perspective dimensions are integrated into the initial perspective feature information to generate reference perspective feature information.

[0124] By performing multi-dimensional processing on the initial perspective feature information based on the perspective dimension parameters after the perspective management operation, images from multiple perspectives can be fully utilized for training, thereby improving the accuracy of the image processing model when processing images.

[0125] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective transformation layer may include at least one perspective transformation matrix and at least one perspective transformation parameter;

[0126] Furthermore, the reference viewing angle feature information is input into the viewing angle transformation layer to generate target viewing angle feature information, including:

[0127] Target perspective feature information is obtained according to reference perspective feature information, each perspective transformation matrix, and each perspective transformation parameter, wherein each perspective transformation matrix and each perspective transformation parameter are generated by each perspective dimension information.

[0128] In practical applications, a perspective transformation matrix and perspective transformation parameters are generated based on the dimensional information included in the reference perspective feature information, and target perspective feature information is generated based on the perspective transformation matrix and transformation parameters.

[0129] Specifically, the perspective transformation matrix is ​​generated based on the dimensional information in the reference perspective feature information, and the perspective transformation parameters are also generated based on the dimensional information in the reference perspective feature information. Furthermore, the perspective transformation matrix and perspective transformation parameters can be understood as processing image feature information based on multiple perspective dimensions so that the image feature information includes more perspective feature information. Consequently, a single perspective image can be processed to include information from multiple perspective dimensions.

[0130] By processing image feature information based on multiple perspective dimensions, the image feature information contains more perspective feature information, and thus a single perspective image can contain information from more dimensions of perspective after processing, so that images from multiple perspectives can be fully utilized for training, thereby improving the accuracy of the image processing model when processing images.

[0131] Specifically, inputting the first-view image feature information and the second-view image feature information into the basic module to obtain a predicted image processing result includes:

[0132] Inputting the first-view image feature information and the second-view image feature information into a second coding layer to obtain mixed-view image feature information;

[0133] Inputting the mixed-view image feature information into a target coding layer set to obtain target coding image feature information, wherein the target coding layer set includes an integer number of coding layers;

[0134] The target coded image feature information is input into the decoder to obtain a predicted image processing result.

[0135] In practical applications, the second-perspective image feature information and the first-perspective image feature information generated by the perspective generation module can be input into the second coding layer to obtain mixed perspective feature information, and then the mixed perspective feature information can be input into the target coding layer set to obtain target coding image feature information, and the obtained target coding image feature information can be input into the decoder to obtain the image processing results required for the image processing task. The image processing results specifically include images with specific target identifiers, images with classification identifiers, and so on.

[0136] Specifically, the second coding layer is a coding layer that inputs the first-perspective image feature information and the second-perspective image feature information, and does not specifically refer to a coding layer at a certain position. The target coding layer set is a coding layer set including an integer number of coding layers. Furthermore, the second coding layer can be understood as combining the first-perspective image feature information and the second-perspective image feature information to generate a mixed perspective feature information that combines perspective features of multiple dimensions. The target coding layer set can be understood as setting multiple coding layers to deepen the application of the image processing model to the obtained mixed perspective feature information.

[0137] In one embodiment provided in this specification, the target coding layer set is set to 2 layers and the model of the basic module is the SegFormer model, then the first perspective information is set to the Q tensor, the second perspective information is set to the K tensor and the V tensor, the three tensors Q, K, and V are input into the second coding layer to obtain mixed perspective image feature information, and then the mixed perspective image feature information is input into the first coding layer in the target coding layer set to obtain an intermediate image feature matrix, and then the intermediate image feature matrix is ​​input into the next coding layer in the target coding layer set to obtain the target coding image feature information. Since the model of the basic module is the SegFormer model, the decoder is configured as a SegFormer model decoder, and the target coding image feature information is input into the SegFormer model decoder to obtain an image with a specific target identifier corresponding to the sample image.

[0138] The first-perspective information and the second-perspective information enhance the image processing model's ability to acquire information about images from multiple different perspectives, thereby making full use of images from multiple perspectives for training, and thereby improving the accuracy of the image processing model in processing images.

[0139] Step 206: Calculate a model loss value based on the predicted image processing result and the sample image processing result.

[0140] After obtaining the processing results of the image processing model, the model loss value can be calculated based on the image processing results and the sample processing results. In the methods provided in this specification, there are many methods for calculating the model loss value, such as the cross entropy loss function, the maximum loss function, the average loss function, etc. In this specification, the specific method of the loss function is not limited, and the actual application shall prevail.

[0141] Step 208: Adjust the model parameters of the image processing model according to the model loss value, and continue training the image processing model until the model training stop condition is reached.

[0142] After obtaining the model loss value, the model parameters of the image processing model can be adjusted according to the model loss value. Specifically, the model loss value can be back-propagated to update the model parameters of the image processing model in sequence.

[0143] After adjusting the model parameters, the above steps can be repeated to continue training the image processing model until the training stop condition is reached. In practical applications, the training stop condition of the image processing model includes:

[0144] The model loss value is less than a preset threshold; and / or

[0145] The training rounds reach the preset training rounds.

[0146] Specifically, during the training of the image processing model, the training stop condition of the model can be set to that the model loss value is less than a preset threshold, or the training stop condition can be set to that the training rounds are preset training rounds, for example, 10 rounds of training. In this specification, there is no specific limitation on the preset threshold of the loss value and / or the preset training rounds, which shall be subject to actual application.

[0147] In addition, after the model training stopping condition is reached, the image processing model training method further includes:

[0148] The viewing angle generation module is removed from the image processing model.

[0149] In practical applications, after the image processing model training is completed, only the basic module can be retained to perform model inference operations.

[0150] Specifically, the perspective generation module is a module that performs a perspective generation operation on the feature information of the first perspective image to generate feature information of the second perspective image. Furthermore, the perspective generation module can be understood as the operation in the perspective generation module can enhance the ability of the image processing model to establish connections between images of multiple perspectives, and the perspective generation module does not contain any neural network structure, and then after the model training is completed, only the trained basic module can be used to perform the image processing process.

[0151] By using the perspective generation module when training the model, the image processing model's ability to establish connections between images from multiple perspectives is enhanced, thereby achieving the accuracy of the image processing model when processing images from multiple perspectives. In addition, when performing image processing tasks, the perspective generation module is eliminated and only the basic module trained based on the perspective generation module is used, further increasing the efficiency of image processing when using the above-mentioned image processing model.

[0152] In addition, after the model training stopping condition is reached, the image processing model training method further includes:

[0153] Obtain model parameters of the image processing model, and send the model parameters of the image processing model to the end-side device.

[0154] In actual applications, the sample images and sample image processing results provided by the client can be received to train the above-mentioned image processing model in the cloud, and the image processing model parameters trained based on the sample images and sample image processing results uploaded by the client can be sent to the client.

[0155] By sending image processing model parameters to the client, a cloud-based model training service can be provided to apply the client's model training method on the cloud, thereby saving the resources required by the client.

[0156] By applying the solution of the above embodiment, the sample image is input into the embedding layer and the encoder set to generate first-perspective image feature information. The generated first-perspective image feature information has the initial features of the sample image, which is convenient for subsequent processing of the sample image; the first-perspective image feature information is then input into the perspective generation module to obtain second-perspective image feature information. The second-perspective image feature information generated by the perspective generation module can be understood as the fusion of multiple-dimensional perspective feature information into second-perspective image feature information based on the original first-perspective image feature information of the sample image, so as to facilitate subsequent processing of the sample image; the first-perspective information and the second-perspective information are then combined for image prediction. The purpose of this step is to combine the original first-perspective image feature information of the sample image with the second-perspective image feature information having multiple-dimensional perspective features, so as to achieve the fusion of the sample image with the characteristic information of multiple perspective dimensions, and then when training the image processing model, the trained model can link the perspective information of multiple dimensions; thereby, the image processing model can make full use of images of multiple perspectives for training, and further, the image processing model trained using the image processing model training method can improve the accuracy of image processing.

[0157] The following further describes the target object recognition method by taking the application of the image processing method provided in this specification to the recognition of a target object as an example, in conjunction with FIG4 . FIG4 shows a process flow chart of a target object recognition method provided in one embodiment of this specification, which specifically includes the following steps:

[0158] Step 402: Acquire the image to be processed collected by the drone.

[0159] In practical applications, a video segment containing multiple images captured by a drone can be obtained, and key frames can be extracted as images to be processed.

[0160] Specifically, the image to be processed is a key frame of a video captured by a drone. Furthermore, the image to be processed can be any color image or grayscale image, a video clip composed of multiple images, and the like.

[0161] In one embodiment provided in this specification, a video clip collected by no one is received and a key frame in the clip is obtained, and the key frame is regarded as an image to be processed.

[0162] Obtaining the images to be processed transmitted by the drone facilitates subsequent image processing.

[0163] Step 404: Input the image to be processed into the image processing model to obtain the water surface area and the target object area in the image to be processed output by the image processing model, and the image processing model is obtained by training the image processing model training method provided in the embodiments of this specification.

[0164] In practical applications, the image to be processed can be input into the image processing model trained in the above-mentioned image processing model training method to obtain the water surface area and the target object area in the image to be processed.

[0165] Specifically, the water surface area is all image areas in the image marked as water surfaces by the image processing model, and the target object area is all image areas in the image marked as target objects by the image processing model, where the target objects can be humans, buildings, vehicles, etc.

[0166] In one embodiment provided in this specification, the image to be processed is input into the above-mentioned image processing model to obtain an image with marked water surface area and image area, and the water surface area and image area in the image are extracted to facilitate subsequent operations.

[0167] The water surface area and the image area in the image are extracted to facilitate subsequent processing of the water surface area and the target object area.

[0168] Step 406: Generate a shore area based on the water surface area.

[0169] In practical applications, the shore area can be generated based on the water surface area processing.

[0170] Specifically, the shore area is a corresponding area generated according to the water surface area. Furthermore, the shore area can be understood as the water surface area expanded and then the original water surface area subtracted.

[0171] The shore area is obtained through the water surface area, so as to subsequently obtain the target object contained in the shore area.

[0172] Specifically, generating a shore area according to the water surface area includes:

[0173] Determining an initial water surface area based on a preset water surface threshold and the water surface area;

[0174] Based on the initial water surface area, a shore area is generated.

[0175] In practical applications, a water surface area that meets the conditions can be selected according to a preset water surface threshold, and then the shore area can be determined based on the water surface area that meets the conditions.

[0176] Specifically, the water surface threshold is a value preset by those skilled in the art when using this method. Furthermore, the preset water surface threshold can be understood as screening out water surfaces that meet the water surface area size. For example, if the threshold is set to 10, then water surfaces with an area of ​​not less than 10m2 will be screened out in the image. 2 The water surface is used as the initial water surface area

[0177] In one embodiment provided in this specification, the preset water surface threshold is 5, and the image processing results above 5m are selected. 2 The water surface is taken as the initial water surface area, and the shore area is generated according to the selected initial water surface area.

[0178] By screening out the water surface areas that meet the target range, errors in the target objects obtained in the final result are avoided.

[0179] Furthermore, based on the initial water surface area, generating a shore area includes:

[0180] Performing polygonal approximation processing on the initial water surface area to generate a reference water surface area;

[0181] Performing polygon expansion processing on the reference water surface area to generate a target water surface area;

[0182] The shore area is generated according to the difference between the reference water surface area and the target water surface area.

[0183] In practical applications, the initial water surface area can be processed by polygonal approximation to generate a reference water surface area, and then the reference water surface area can be processed by polygonal expansion to generate a target water surface area. Finally, the shore area is generated according to the difference between the reference water surface area and the target water surface area.

[0184] Specifically, polygon approximation represents the target area using a polygon with fewer vertices, while polygon expansion updates the vertices of the polygons according to the polygon expansion algorithm. Furthermore, polygon approximation followed by expansion can be understood as expanding the original water surface area.

[0185] It should be noted that the existing method of expanding the area of ​​a water surface area usually involves first binarizing the image to obtain the target to be expanded. Morphological expansion is to expand the boundary points of the binary object, merge all background points that are in contact with the object into the object, and expand the boundary outward. If two objects are close together, the two objects will be connected together, and the expanded object boundaries of different sizes will be obtained according to different expansion distances. The calculation steps required by this method are cumbersome and place great pressure on the computer. The polygonal approximation and polygonal expansion algorithms provided in this manual greatly reduce the number of calculation steps.

[0186] In one embodiment provided in this specification, an initial water surface area represented by 100 coordinate points is approximated and represented by 10 vertices, and then a polygon expansion algorithm is used on the approximated reference water surface area, as shown in Formula Group 1:

[0187] Among them, P i is the vertex after expansion, Q i is the original vertex, L is the expansion (scaling) distance, and M is the vector The size of and is a vector The component vectors of θ are and The angle between , Normalize(v1) and Normalize(v2) respectively represent and The unit vector of .

[0188] The shore area is obtained by subtracting the expanded target water surface area from the initial water surface area.

[0189] By using polygons with fewer vertices to represent the initial water surface and approximating the water surface area using polygon expansion, the computing resources required to expand the water surface area are greatly reduced. The shore area is then confirmed by the difference between the expanded water surface area and the initial water surface area, which improves the efficiency of calculating the shore area based on the water surface area, thereby improving the efficiency of confirming the target object.

[0190] Step 408: Determine the recognition result of the target object based on the comparison result between the target object area and the shore area.

[0191] In practical applications, the target object area and the shore area can be compared, and the recognition result of the target object can be determined based on the comparison result.

[0192] Specifically, the target object recognition result is to mark out target objects that meet the requirements, such as marking out buildings on the shore, marking out cars parked on the shore, etc.

[0193] In one embodiment provided in this specification, a cross-validation calculation is performed on the target object area and the shore area to obtain the target object on the shore and mark it in the image.

[0194] By inputting the to-be-processed image acquired by the UAV into the above-mentioned image processing model, the recognition result of the target object is obtained according to the image processing result output by the image processing model, thereby achieving accurate automatic recognition of the target object.

[0195] The solution of the above embodiment is applied, and the image collected by the drone is input into the image processing model trained according to the image processing model training method provided in the embodiment of this specification to obtain images of the marked water surface area and the target object area. The application of the above image processing model ensures the accuracy of processing the image collected by the drone; then, the water surface area to be expanded is selected according to the preset threshold, which effectively avoids expanding the water surface area that does not need to be processed and reduces the unnecessary performance overhead when expanding the water surface area; when expanding the water surface area, the water surface area is first polygonally approximated, and then polygon expansion is performed based on the approximated polygons. Since the water surface area is multivariate approximated, the amount of calculation when performing polygon expansion is greatly reduced. At the same time, the polygon expansion algorithm used when expanding the polygon greatly improves the efficiency of the operation compared to the morphological dilation algorithm based on the water surface area; the shore area is subtracted from the original water surface area after expansion, and the desired target object is obtained based on the relationship between the obtained shore area and the target object area, thereby realizing automatic detection of the target object, and thus enabling the drone to quickly and automatically identify the target object.

[0196] The following, in conjunction with Figure 5, takes the application of the image processing model training method provided in this specification in the semantic segmentation model training as an example to further illustrate the semantic segmentation model training method. Figure 5 shows a processing flow chart of a semantic segmentation model training method provided in one embodiment of this specification, which specifically includes the following steps:

[0197] Step 502: Obtain a sample image and a segmentation result corresponding to the sample image.

[0198] Step 504: Input the sample image into the embedding layer of the image processing model to obtain sample image coding information.

[0199] Step 506: Input the sample image coding information into the first coding layer in the basic module to obtain the first-view image feature information, wherein the basic module contains 4 coding layers.

[0200] Step 508: Input the first-view image feature information into the view extraction layer in the view generation module, and obtain initial view feature information through the view encoding unit in the view extraction layer.

[0201] Step 510: Input the initial viewing angle feature information into the viewing angle management layer to obtain reference viewing angle feature information.

[0202] Step 512: Input the reference viewing angle feature information into the viewing angle transformation layer to obtain the target viewing angle feature information.

[0203] Step 514: input the target perspective feature information into the perspective extraction layer, and obtain the second perspective image feature information through the perspective decoding unit of the perspective extraction layer.

[0204] Step 516: Input the first-view image feature information and the second-view image feature information into the second coding layer to obtain the first image feature information.

[0205] Step 518: Input the first image feature information into the third coding layer to obtain the second image feature information.

[0206] Step 520: Input the second image feature information into the fourth coding layer to obtain third image feature information.

[0207] Step 522: Input the one-view image feature information, the first image feature information, the second image feature information, and the third image feature information into a decoding layer in a decoder to obtain sample image feature information.

[0208] Step 524: Input the sample image feature information into the image segmentation layer in the decoder to obtain the predicted sample image segmentation result.

[0209] Step 526: Calculate the loss value of the model prediction result by combining the predicted sample image segmentation result and the segmentation result corresponding to the sample image.

[0210] Step 528: Adjust the parameters of the image processing model according to the obtained prediction loss value.

[0211] By applying the solution of the above embodiment, the sample image is input into the embedding layer and the encoder set to generate first-perspective image feature information. The generated first-perspective image feature information has the initial features of the sample image, which is convenient for subsequent processing of the sample image; the first-perspective image feature information is then input into the perspective generation module to obtain the second-perspective image feature information. The second-perspective image feature information generated by the perspective generation module can be understood as the fusion of multiple-dimensional perspective feature information into second-perspective image feature information based on the original first-perspective image feature information of the sample image, so as to facilitate subsequent processing of the sample image; the first-perspective information and the second-perspective information are then combined for image prediction. The purpose of this step is to combine the original first-perspective image feature information of the sample image with the second-perspective image feature information with multiple-dimensional perspective features, so as to achieve the fusion of the sample image with the characteristic information of multiple perspective dimensions, and then when training the image processing model, the trained model can link the perspective information of multiple dimensions; thereby, the image processing model can make full use of images of multiple perspectives for training, and further, the image processing model trained using the image processing model training method can improve the accuracy of image processing.

[0212] With the rapid development of drone technology, drones have been widely used in research fields and industrial applications. Images and videos are important ways for drones to perceive their surroundings. Drones are often in motion during flight, which causes the perspective of images collected by drones to constantly change, thereby making the accuracy of existing image processing methods applied to drones lower than expected. Based on this, one or more specific implementations provided in this specification include an image processing method. Specifically, the following, in conjunction with Figure 6, takes the application of the image processing method provided in this specification in the identification of target objects as an example to further illustrate the image processing method. Among them, Figure 6 shows a processing flow chart of a target object identification method provided in an embodiment of this specification, which specifically includes the following steps:

[0213] Step 602: Receive an image processing request sent by a user, wherein the image processing request carries an image to be processed.

[0214] In actual applications, it is possible to receive an image processing request sent by a user and obtain the image to be processed carried in the request.

[0215] Specifically, the image processing request includes the image processing task type and the image to be processed. The image to be processed can be any color image or grayscale image, a video clip composed of multiple images, etc. The task type can be a semantic segmentation task, an image classification task, etc.

[0216] In one example provided in this specification, a user inputs a video clip captured by a drone and requests to obtain a feature image containing possible ruins, relics, and geographical features. The task type requested by the user is determined to be an image classification task, and the output image processing result is a feature image classified as containing possible ruins, relics, and geographical features. Key extraction is performed on the video clip input by the user to obtain the image to be processed.

[0217] The image to be processed in the request is obtained so that the image to be processed is processed using the image processing model described below.

[0218] Step 604: Input the image to be processed into an image processing model to obtain an image processing result output by the image processing model, wherein the image processing model is obtained by training using the image processing model training method provided in the embodiments of this specification.

[0219] In practical applications, the acquired image to be processed can be input into an image processing model trained by the image processing model training method provided by the embodiments of this specification to obtain an image processing result.

[0220] In one embodiment provided in this specification, key frame images extracted from video clips are input into the image processing model completed for training to obtain the classification of each key frame image.

[0221] By inputting the image to be processed into the image processing model trained by the image processing model training method provided in the embodiments of this specification, the image processing result can be obtained so that images containing multiple perspectives can be processed with high precision.

[0222] Step 606: Send the image processing result to the user.

[0223] In practical applications, the image processing results can be sent to users.

[0224] In one embodiment provided in this specification, feature images classified as containing possible ruins, relics and geographical features are selected and sent to the user based on the classification results output by the above-mentioned image processing model.

[0225] By processing the image processing task input by the user using the image processing model trained based on the image processing model training method provided in the embodiments of this specification, high-precision processing of images containing multiple perspectives is achieved.

[0226] By applying the solution of the above-mentioned embodiment, it is possible to accept image processing requests sent by users, and to process the received image processing requests using the image processing model trained by the image processing model training method provided in the embodiments of this specification, and to return the processing results to the user. Since the image processing model trained by the image processing model training method provided in the embodiments of this specification can accurately process images, the image processing method can accurately process image processing requests sent by users.

[0227] Corresponding to the above method embodiments, this specification also provides an embodiment of an image processing model training device. FIG7 shows a schematic diagram of the structure of an image processing model training device provided in one embodiment of this specification. As shown in FIG7 , the device includes:

[0228] An acquisition module 702 acquires a sample image and a sample image processing result corresponding to the sample image;

[0229] A prediction module 704 inputs the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-view image feature information and second-view image feature information, where the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information;

[0230] A judgment module 706 calculates a model loss value according to the predicted image processing result and the sample image processing result;

[0231] The stopping module 708 adjusts the model parameters of the image processing model according to the model loss value, and continues to train the image processing model until the model training stopping condition is reached.

[0232] Optionally, the image processing model includes a basic module and a perspective generation module;

[0233] The prediction module 704 is further configured to:

[0234] Inputting the sample image into the basic module to obtain first-view image feature information;

[0235] Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information;

[0236] The first-view image feature information and the second-view image feature information are input into the basic module to obtain a predicted image processing result.

[0237] Optionally, the basic module includes an encoder and a decoder;

[0238] The prediction module 704 is further configured to:

[0239] The sample image is input into the encoder to obtain first-view image feature information.

[0240] Optionally, the encoder includes an embedding layer and at least two encoding layers;

[0241] The prediction module 704 is further configured to:

[0242] Inputting the sample image into the embedding layer to obtain sample image encoding information;

[0243] Inputting the sample image coding information into an initial coding layer set to obtain initial coding image feature information, wherein the initial coding layer set includes an integer number of coding layers;

[0244] The initial coded image feature information is input into the first coding layer to obtain the first-view image feature information.

[0245] Optionally, the perspective generation module includes a perspective extraction layer, a perspective management layer, and a perspective transformation layer;

[0246] The prediction module 704 is further configured to:

[0247] Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information;

[0248] Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information;

[0249] Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information;

[0250] The target perspective feature information is input into the perspective extraction layer to generate second perspective image feature information.

[0251] Optionally, the view extraction layer includes a view encoding unit and a view decoding unit;

[0252] The prediction module 704 is further configured to:

[0253] Inputting the first viewing angle image information into the viewing angle encoding unit to generate initial viewing angle feature information;

[0254] Accordingly, the target perspective feature information is input into the perspective extraction layer to generate second perspective image feature information, including:

[0255] The target viewing angle feature information is input into the viewing angle decoding unit to generate second viewing angle image feature information.

[0256] Optionally, the viewing angle management layer includes at least one viewing angle dimension information;

[0257] The prediction module 704 is further configured to:

[0258] Reference viewing angle feature information is generated according to the initial viewing angle feature information and each viewing angle dimension information.

[0259] Optionally, the perspective transformation layer includes at least one perspective transformation matrix and at least one perspective transformation parameter, wherein each perspective transformation matrix and each perspective transformation parameter are generated by each perspective dimension information;

[0260] The prediction module 704 is further configured to:

[0261] Target view feature information is obtained according to the reference view feature information, each view transformation matrix, and each view transformation parameter.

[0262] Optionally, the prediction module 704 is further configured to:

[0263] Inputting the first-view image feature information and the second-view image feature information into a second coding layer to obtain mixed-view image feature information;

[0264] Inputting the mixed-view image feature information into a target coding layer set to obtain target coding image feature information, wherein the target coding layer set includes an integer number of coding layers;

[0265] The target coded image feature information is input into the decoder to obtain a predicted image processing result.

[0266] Optionally, the image processing model training device further includes:

[0267] A removal module removes the perspective generation module from the image processing model after the model training stop condition is reached.

[0268] Optionally, the image processing model training device further includes:

[0269] The sending module obtains the model parameters of the image processing model after the model training stop condition is met, and sends the model parameters of the image processing model to the end-side device.

[0270] The above is a schematic diagram of an image processing model training device according to this embodiment. It should be noted that the technical solution of this image processing model training device and the technical solution of the aforementioned image processing model training method are based on the same concept. For any details not described in detail in the technical solution of the image processing model training device, please refer to the description of the technical solution of the aforementioned image processing model training method.

[0271] Figure 8 shows a block diagram of a computing device 800 according to one embodiment of this specification. Components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0272] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0273] In one embodiment of the present specification, the aforementioned components of computing device 800 and other components not shown in FIG8 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG8 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0274] Computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 800 may also be a mobile or stationary server.

[0275] Among them, the processor 820 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0276] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the aforementioned image processing model training method, image processing method, or target object recognition method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned image processing model training method, image processing method, or target object recognition method.

[0277] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0278] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above-mentioned image processing model training method, image processing method, or target object recognition method are based on the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-mentioned image processing model training method, image processing method, or target object recognition method.

[0279] One embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0280] The above is a schematic diagram of a computer program according to this embodiment. It should be noted that the technical solution of this computer program is based on the same concept as the technical solution of the image processing model training method, image processing method, or target object recognition method described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the image processing model training method, image processing method, or target object recognition method described above.

[0281] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0282] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0283] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0284] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0285] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training an image processing model, comprising: Acquire a sample image and a sample image processing result corresponding to the sample image; Input the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated according to first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information; Calculate the model loss value according to the predicted image processing result and the sample image processing result; The model parameters of the image processing model are adjusted according to the model loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

2. The method according to claim 1, wherein the image processing model comprises a basic module and a perspective generation module; Inputting the sample image into an image processing model to obtain a predicted image processing result output by the image processing model includes: Inputting the sample image into the basic module to obtain first-view image feature information; Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information; The first-view image feature information and the second-view image feature information are input into the basic module to obtain a predicted image processing result.

3. The method of claim 2, wherein the basic module comprises an encoder and a decoder, wherein the encoder comprises an embedding layer and at least two encoding layers; Inputting the sample image into the basic module to obtain first-view image feature information includes: Inputting the sample image into the embedding layer to obtain sample image encoding information; Inputting the sample image coding information into an initial coding layer set to obtain initial coding image feature information, wherein the initial coding layer set includes an integer number of coding layers; The initial encoded image feature information is input into the first encoding layer to obtain the first viewing angle image feature information.

4. The method according to claim 2, wherein the view generation module comprises a view extraction layer, a view management layer and a view transformation layer; Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information includes: Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information; Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information; Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information; The target viewing angle feature information is input into the viewing angle extraction layer to generate second viewing angle image feature information.

5. The method of claim 4, wherein the view extraction layer comprises a view encoding unit and a view decoding unit; Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information includes: Inputting the first viewing angle image information into the viewing angle encoding unit to generate initial viewing angle feature information; Accordingly, the target viewing angle feature information is input into the viewing angle extraction layer to generate the second viewing angle image feature information, including: The target viewing angle feature information is input into the viewing angle decoding unit to generate second viewing angle image feature information.

6. The method according to claim 4, wherein the viewing angle management layer comprises at least one viewing angle dimension information; Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information includes: Reference viewing angle feature information is generated according to the initial viewing angle feature information and each viewing angle dimension information.

7. The method of claim 6, wherein the perspective transformation layer comprises at least one perspective transformation matrix and at least one perspective transformation parameter, wherein: Each viewing angle transformation matrix and each viewing angle transformation parameter are generated from each viewing angle dimension information; Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information includes: The target viewing angle feature information is obtained according to the reference viewing angle feature information, each viewing angle transformation matrix and each viewing angle transformation parameter.

8. The method according to claim 3, inputting the first-view image feature information and the second-view image feature information into the basic module to obtain a predicted image processing result, comprising: Inputting the first-view image feature information and the second-view image feature information into a second coding layer to obtain mixed-view image feature information; Inputting the mixed-view image feature information into a target coding layer set to obtain target coding image feature information, wherein the target coding layer set includes an integer number of coding layers; The target encoded image feature information is input into the decoder to obtain a predicted image processing result.

9. An image processing method, comprising: Receiving an image processing request sent by a user, wherein the image processing request carries an image to be processed; Inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model, wherein the image processing model is obtained by the method according to any one of claims 1 to 8; The image processing result is sent to the user.

10. A method for identifying a target object, comprising: Obtain the images to be processed collected by the drone; The image to be processed is input into the image processing model to obtain the image to be processed output by the image processing model. The water surface area and the target object area in the image, the image processing model is obtained by the method according to any one of claims 1 to 8; generating a shore area according to the water surface area; The recognition result of the target object is determined according to the comparison result between the target object area and the shore area.

11. The method according to claim 10, generating a shore area according to the water surface area, comprising: Determining an initial water surface area based on a preset water surface threshold and the water surface area; Based on the initial water surface area, a shore area is generated.

12. The method according to claim 11, generating a shore area based on the initial water surface area, comprising: Performing polygonal approximation processing on the initial water surface area to generate a reference water surface area; Performing polygon expansion processing on the reference water surface area to generate a target water surface area; The shore area is generated according to the difference between the reference water surface area and the target water surface area.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the image processing model training method described in any one of claims 1 to 8, the image processing method described in claim 9, or the target object recognition method described in any one of claims 10 to 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the image processing model training method described in any one of claims 1 to 8, the image processing method described in claim 9, or the target object recognition method described in any one of claims 10 to 12.

Images