Image processing model training method and image processing method

Abstract

The embodiment of the invention provides an image processing model training method and an image processing method. The image processing model training method comprises the following steps: acquiring a sample image and a sample image processing result corresponding to the sample image; inputting the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, the predicted image processing result being generated according to first view image feature information and second view image feature information, the first view image feature information being generated based on the sample image, and the second view image feature information being generated based on the sample image; the second visual angle image feature information is generated based on the first visual angle image feature information; calculating a model loss value according to the predicted image processing result and the sample image processing result; and adjusting model parameters of the image processing model according to the model loss value and continuing to train the image processing model until a model training stop condition is reached. Therefore, the image processing model can fully utilize the images of multiple visual angles for training, so that the accuracy of the image processing model during image processing is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to an image processing model training method. Background Art

[0002] With the rapid development of drone technology, drones have been widely used in research and industrial applications. Images and videos are important ways for drones to perceive their surroundings. Image semantic segmentation is a research hotspot in the field of computer vision and is widely used in scenarios such as unmanned driving and intelligent robots. Semantic segmentation of drone aerial images uses semantic segmentation technology to enable drones to acquire intelligent perception capabilities of scene targets based on drone aerial images.

[0003] However, drones are often in motion during flight, which causes the image perspectives collected by drones to change constantly, making the accuracy of existing image processing methods applied to drones lower than expected. Therefore, those skilled in the art need a more accurate image processing method in drone scenarios. Summary of the invention

[0004] In view of this, the embodiments of this specification provide an image processing model training method, an image processing method, and a target object recognition method. One or more embodiments of this specification also involve an image processing model training device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an image processing model training method, comprising:

[0006] Acquire a sample image and a sample image processing result corresponding to the sample image;

[0007] Input the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated according to first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information;

[0008] Calculate the model loss value according to the predicted image processing result and the sample image processing result;

[0009] The model parameters of the image processing model are adjusted according to the model loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

[0010] According to a second aspect of the embodiments of this specification, there is provided an image processing method, including:

[0011] Receiving an image processing request sent by a user, wherein the image processing request carries an image to be processed;

[0012] Inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model, wherein the image processing model is obtained by the image processing model training method described in the first aspect of the above embodiment;

[0013] The image processing result is sent to the user.

[0014] According to a third aspect of an embodiment of this specification, a method for identifying a target object is provided, including:

[0015] Obtain the images to be processed collected by the drone;

[0016] Inputting the image to be processed into an image processing model to obtain a water surface area and a target object area in the image to be processed output by the image processing model, wherein the image processing model is obtained by the image processing model training method described in the first aspect of the above embodiment;

[0017] generating a shore area according to the water surface area;

[0018] The recognition result of the target object is determined according to the comparison result between the target object area and the shore area.

[0019] According to a fourth aspect of the embodiments of this specification, there is provided an image processing model training device, comprising:

[0020] An acquisition module, which acquires a sample image and a sample image processing result corresponding to the sample image;

[0021] A prediction module, inputting the sample image into an image processing model, and obtaining a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated according to first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information;

[0022] A judgment module, which calculates a model loss value according to the predicted image processing result and the sample image processing result;

[0023] The stopping module adjusts the model parameters of the image processing model according to the model loss value, and continues to train the image processing model until the model training stopping condition is reached.

[0024] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0025] Memory and processor;

[0026] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned image processing model training method or image processing method or target object recognition method are implemented.

[0027] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0028] According to the seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0029] An image processing model training method provided by an embodiment of the present specification obtains a sample image and a sample image processing result corresponding to the sample image; inputs the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information; calculates a model loss value based on the predicted image processing result and the sample image processing result; adjusts model parameters of the image processing model based on the model loss value, and continues to train the image processing model until a model training stop condition is reached.

[0030] According to the method provided in the embodiments of this specification, first-perspective image feature information is generated according to the sample image, and then second-perspective image feature information is obtained according to the first-perspective image feature information, and the image processing result is predicted according to the first-perspective image feature information and the second-perspective image feature information, and the first-perspective image feature information and the second-perspective image feature information are combined for image prediction. This can be understood as converting the sample image features that originally only had a single perspective into image features with multiple perspectives, and then combining the original single-perspective image features with the multiple-perspective image features and inputting them into the next layer of the model. This can enable the image processing model to fully utilize images of multiple perspectives for training, and thus improve the accuracy of the image processing model trained using the image processing model training method when processing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is an architecture diagram of an image processing model training system provided by an embodiment of this specification;

[0032] Figure 2 is a flowchart of an image processing model training method provided by an embodiment of this specification;

[0033] Figure 3 is a structural diagram of an image processing model provided by an embodiment of this specification;

[0034] Figure 4 is a process flow chart of a target object identification method provided by an embodiment of this specification;

[0035] Figure 5 is a processing flow chart of a semantic segmentation model training method provided by an embodiment of this specification;

[0036] Figure 6 is a flow chart of an image processing method provided by an embodiment of this specification;

[0037] Figure 7 It is a structural schematic diagram of an image processing model training device provided by an embodiment of this specification;

[0038] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0039] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0040] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0042] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] First, the terms involved in one or more embodiments of this specification are explained.

[0044] ViT (Vision Transformer): An image processing model based on the Transformer architecture. Initially, the Transformer model was mainly used for natural language processing tasks, but in recent years, researchers have found that it also performs well in computer vision tasks. The ViT model divides the image into multiple small blocks, flattens these small blocks into one-dimensional vectors, and then inputs these vectors into the Transformer model for processing.

[0045] SETR (SEgmentation Transformer): A semantic segmentation model based on the Transformer architecture.

[0046] SegFormer: A Transformer-based semantic segmentation model that combines Transformer with convolutional neural networks to achieve more efficient and accurate image segmentation.

[0047] Morphological Expansion: Morphological expansion is a method in image processing. Before expansion, the image is first binarized to obtain the target to be expanded. Morphological expansion is to expand the boundary points of the binary object, merge all the background points that are in contact with the object into the object, and expand the boundary outward. If the two objects are close to each other, the two objects will be connected together. Different sizes of expanded object boundaries are obtained according to different expansion distances.

[0048] In this specification, an image processing model training method is provided. This specification also relates to an image processing model training device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0049] See also Figure 1 , Figure 1 An architecture diagram of an image processing model training system provided by an embodiment of the present specification is shown. The image processing model training system may include a client 100 and a server 200;

[0050] The client 100 is used to send a sample image and a sample image processing result corresponding to the sample image to the server 200;

[0051] The server 200 is used to input the sample image into the image processing model, obtain the predicted image processing result output by the image processing model, wherein the predicted image processing result is generated according to the first-view image feature information and the second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information; calculate the model loss value according to the predicted image processing result and the sample image processing result; adjust the model parameters of the image processing model according to the model loss value, and continue to train the image processing model until the model training stop condition is reached; send the model parameters of the image processing model to the client 100;

[0052] The client 100 is also used to receive the model parameters of the image processing model sent by the server 200.

[0053] By applying the scheme of the embodiments of the present specification, first-perspective image feature information is generated based on the sample image, and then second-perspective image feature information is obtained based on the first-perspective image feature information, and image processing results are predicted based on the first-perspective image feature information and the second-perspective image feature information, and the first-perspective image feature information and the second-perspective image feature information are combined for image prediction. This can be understood as converting the sample image features that originally only had a single perspective into image features with multiple perspectives, and then combining the original single-perspective image features with the multiple-perspective image features and inputting them into the next layer of the model. This can enable the image processing model to make full use of images from multiple perspectives for training, and thus improve the accuracy of the image processing model trained using the image processing model training method when processing images.

[0054] The image processing system may include multiple clients 100 and a server 200, wherein the client 100 may be referred to as a client-side device and the server 200 may be referred to as a cloud-side device. Multiple clients 100 may establish a communication connection through the server 200. In the image processing scenario, the server 200 is used to provide image processing services between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver, respectively, and realize communication through the server 200.

[0055] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the image processing scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates an image processing result according to the data stream and pushes the image processing result to other clients that have established communication.

[0056] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0057] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device, and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that provide support for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0059] It is worth noting that the image processing model training method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the image processing model training method provided in the embodiments of this specification. In other embodiments, the image processing model training method provided in the embodiments of this specification may also be jointly executed by the client and the server.

[0060] See also Figure 2 , Figure 2 A flowchart of an image processing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0061] Step 202: Obtain a sample image and a sample image processing result corresponding to the sample image.

[0062] In actual applications, the image processing tasks sent by the user can be received through the server or the client.

[0063] Specifically, the training method provided in this specification is supervised training, which includes sample images and sample processing results corresponding to the sample images. Specifically, the sample image is an arbitrary image input, and the corresponding sample processing results are determined by a variety of image processing tasks, including semantic segmentation, key point detection, image generation, etc.

[0064] It should be noted that the sample images include any color images or grayscale images, video clips composed of multiple images, etc. The processing results of the sample images have different results according to the different task classifications corresponding to the image processing. For example, the sample processing result corresponding to the semantic segmentation task is an image with the same size as the sample image, in which each pixel is marked with a category. Common categories include people, cars, roads, buildings, etc. The processing result of key point detection is usually a key point position annotation, which is used to specify the position of a specific key point in the image. Key points can be some specific points, such as facial key points (such as eyes, nose, mouth, etc.), human joints (such as shoulders, elbows, knees, etc.) or other key points. The annotation method can use pixel coordinates or normalized coordinates to represent the position of the key point.

[0065] In one embodiment provided in this specification, the sample image may be an image captured by a drone, and the corresponding image processing result may be an image carrying water surface markers and building markers. The required building sequence is detected by processing the water surface and judging the building.

[0066] The model training is completed by receiving a sample image, obtaining a predicted sample image processing result according to the obtained sample image, comparing the predicted sample image processing result with the obtained sample image processing result, and adjusting the parameters of the model according to the comparison result.

[0067] Step 204: Input the sample image into the image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated based on first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information.

[0068] In practical applications, the sample image is input into the image processing model, and a predicted image processing result is generated according to the first-view image feature information and the second-view image feature information of the sample image.

[0069] Specifically, the first-perspective image feature information is a feature matrix obtained by the image processing model based on the sample image, and the second-perspective image feature is a feature matrix obtained based on the first-perspective image. Furthermore, the first-perspective image feature information can be understood as a feature matrix with basic features of the sample image, which is used for the subsequent image processing model to process, and the second-perspective image feature information can be understood as a feature matrix after processing the first-perspective image feature information, which is used to enhance the image processing model's processing capabilities for multiple images from different perspectives.

[0070] In one implementation provided in this specification, the image processing task can be applied to the application scenario of semantic segmentation, and the position of the key points in the image that meet the annotations can be determined according to the annotations given by the technician. For example, in the application scenario of drone detection of buildings in the shore area, the image areas marked as "water" and "buildings" can be detected based on the multi-view images collected by the drone, and then the buildings in the image belonging to the shore can be confirmed, thereby helping the technicians to distinguish the buildings on the shore for subsequent management.

[0071] For example, in the application scenario of archaeological exploration, drones can take high-resolution surface images at the archaeological site, identify possible ruins, relics and geographical features through image classification technology, and thus assist archaeologists in exploration work.

[0072] In practical applications, the collected image can be any color image or grayscale image, etc.

[0073] By inputting the sample image into the image processing model, a predicted image processing result can be generated based on the first-perspective image feature information and the second-perspective image feature information of the sample image, thereby improving the accuracy of the image processing model when processing images from multiple different perspectives.

[0074] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the image processing model may include a basic module and a perspective generation module;

[0075] Furthermore, the sample image is input into an image processing model to obtain a predicted image processing result output by the image processing model, including:

[0076] Inputting the sample image into the basic module to obtain first-view image feature information;

[0077] Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information;

[0078] The first-view image feature information and the second-view image feature information are input into the basic module to obtain a predicted image processing result.

[0079] See also Figure 3 , Figure 3 FIG. 1 shows a schematic diagram of the structure of an image processing model provided by an embodiment of the present specification, such as Figure 3As shown, the image processing model includes a basic module 302 and a viewing angle generation module 304. In practical applications, the sample image is used to generate first viewing angle image feature information in the basic module 302, and then the first viewing angle image feature information is input into the viewing angle generation module 304 to generate second viewing angle image feature information, and the first viewing angle image feature information and the second viewing angle image feature information are input into the basic module 302 to obtain a predicted image processing result.

[0080] Specifically, the basic module is any image processing model based on the Transformer architecture and including at least two encoder cores, such as ViT, SETR, SegFormer, etc., which is not limited in this specification. The perspective generation module is a module that processes the first perspective feature information to obtain the second perspective information. Furthermore, the basic module can be understood as the basic architecture when using the above-mentioned image processing model to perform model training tasks and image processing tasks, and the perspective generation module can be understood as a module that improves the model's ability to obtain information from multiple perspective images during the training process of the image processing model.

[0081] By adding a perspective generation module during image processing model training, the model's ability to obtain information from multiple perspective images is improved, thereby improving the accuracy of image processing performed by the obtained image processing model.

[0082] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the basic module may include an encoder and a decoder;

[0083] Specifically, the sample image is input into the basic module to obtain the first-view image feature information, including:

[0084] The sample image is input into the encoder to obtain first-view image feature information.

[0085] See also Figure 3 ,like Figure 3 The basic module 302 shown includes an encoder 306 and a decoder 308. In practical applications, a sample image is input into the encoder 306 to obtain first-view image feature information.

[0086] Specifically, the encoder is used to process the sample image into the feature information corresponding to the sample image, and the decoder is used to process the image feature information obtained in the encoder into the information required by the task type. Furthermore, the encoder can be understood as extracting the features of the sample image and processing it according to the parameters of the model, and the decoder can be understood as processing the feature matrix processed by the encoder into the result required by the task type. For example, if the task type is a semantic segmentation task, the role of the decoder is to process the feature matrix processed by the encoder into an image with a specific target identifier. For example, if the task type is an image classification task, the role of the decoder is to process the feature matrix processed by the encoder into an image with a classification identifier.

[0087] By encoding the sample image, the computer can extract the features of the sample image, and by decoding the features, the computer can realize the actual processing of the sample image.

[0088] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the encoder may include an embedding layer and at least two encoding layers;

[0089] Specifically, the sample image is input into the encoder to obtain first-view image feature information, including:

[0090] Inputting the sample image into the embedding layer to obtain sample image encoding information;

[0091] Inputting the sample image coding information into an initial coding layer set to obtain initial coding image feature information, wherein the initial coding layer set includes an integer number of coding layers;

[0092] The initial encoded image feature information is input into the first encoding layer to obtain the first viewing angle image feature information.

[0093] See also Figure 3 ,like Figure 3 The encoder 306 shown includes an embedding layer 310, an initial coding layer set 312, a first coding layer 314, a second coding layer 316, and a target coding layer set 318. In practical applications, the sample image is subjected to the embedding operation of the embedding layer 310 to obtain sample image coding information, the sample image coding information is input into the initial coding layer set to obtain initial coding image feature information, and the initial coding image feature information is input into the first coding layer 314 to obtain first viewing angle feature information.

[0094] Specifically, the embedding layer is an image processing layer that processes the sample image into sample image coding information, the initial coding layer set is a coding layer set containing an integer number of coding layers, and the first coding layer is a coding layer for obtaining first-view feature information, and does not specifically refer to a coding layer at a certain position. Further, the embedding layer can be understood as performing matrix processing on the sample image to facilitate the subsequent coding operation, and the coding layer can be understood as performing a coding operation on the matrixed image after the embedding layer processing, and outputting information with sample image characteristics.

[0095] It should be noted that the embedding layer refers to the matrix operation on the sample image, which is related to the model used by the basic module. For example, if the model used by the basic module is the SegFormer model, then the embedding layer is a lightweight CNN encoder in the SegFormer model to extract the multi-scale features of the image and a word segmentation embedding unit to perform word embedding operations on the multi-scale features extracted by CNN. If the model used by the basic module is the ViT model, then the embedding layer is the word segmentation embedding unit in the ViT model to perform word embedding operations on the sample image.

[0096] In one embodiment provided in this specification, the basic module uses the SegFormer model and sets the output coding layer set to zero, so after the image processing receives the sample image, it first inputs the sample image into a simple CNN encoder to obtain the multi-scale features of the sample image, and inputs the multi-scale features into the word segmentation embedding unit to obtain the sample image encoding information corresponding to the sample image, and the obtained sample image encoding information is input into the first encoder to obtain the first-view image feature information.

[0097] By performing matrix operations on the sample image, it is convenient for the image processing model to process the sample image.

[0098] In order to fully utilize images of multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of the present specification, the perspective generation module may include a perspective extraction layer, a perspective management layer, and a perspective transformation layer;

[0099] Furthermore, inputting the first-view image feature information into the view generation module to obtain the second-view image feature information includes:

[0100] Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information;

[0101] Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information;

[0102] Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information;

[0103] The target viewing angle feature information is input into the viewing angle extraction layer to generate second viewing angle image feature information.

[0104] See also Figure 3 ,like Figure 3 The perspective generation module 304 shown includes a perspective extraction layer 320, a perspective transformation layer 324, and a perspective management layer 322. In practical applications, the first perspective image information can be input into the perspective extraction layer 320 to generate initial perspective feature information, and then the initial perspective feature information can be input into the perspective management layer 322 to generate reference perspective feature information, and then the reference perspective feature information can be input into the perspective transformation layer 324 to generate target perspective feature information, and finally the target perspective feature information can be input into the perspective extraction layer to generate second perspective image feature information.

[0105] Specifically, the perspective extraction layer can encode the first perspective image feature information passed into the perspective generation module into initial perspective feature information through the encoding core, or can decode the target perspective feature information generated in the perspective generation module through the decoding core and output it to the perspective generation module and pass it into the second encoding layer. The perspective management layer can receive the initial perspective feature information encoded by the perspective extraction layer and perform multi-dimensional processing on the received initial perspective feature information to obtain reference target feature information. The perspective transformation layer receives the perspective reference target feature information and obtains the target perspective feature information through dimensional transformation based on the dimensional information of the perspective management layer.

[0106] By performing perspective generation operations on sample images, the image processing model fully utilizes images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images.

[0107] In order to fully utilize images of multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective extraction layer may include a perspective encoding unit and a perspective decoding unit;

[0108] Specifically, inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information includes:

[0109] Inputting the first viewing angle image information into the viewing angle encoding unit to generate initial viewing angle feature information;

[0110] Accordingly, the target viewing angle feature information is input into the viewing angle extraction layer to generate the second viewing angle image feature information, including:

[0111] The target viewing angle feature information is input into the viewing angle decoding unit to generate second viewing angle image feature information.

[0112] See also Figure 3 ,like Figure 3 The perspective extraction layer 320 shown includes a perspective encoding unit 326 and a perspective decoding unit 328. In practical applications, the first perspective image information can be input into the perspective encoding unit 326 to obtain initial perspective feature information for subsequent perspective generation processing, and the target perspective feature information after the perspective generation can also be decoded by the decoding unit 328 to generate the second perspective image information.

[0113] Specifically, the encoding unit is a unit that encodes the first-perspective image information, and the decoding unit is a unit that decodes the target-perspective feature information. Further, the role of the encoding unit can be understood as extracting the core information in the first-perspective image information, compressing the first-perspective image information to facilitate subsequent processing based on the center dimension, and the role of the decoding unit can be understood as restoring the target-perspective feature information generated according to the perspective to the second-perspective image feature information of the same length as the first-perspective image feature information.

[0114] It should be noted that the encoding unit and the decoding unit are both the encoder cores in a Transformer model, and their purpose is to change the length of the received data while ensuring the integrity of the original information as much as possible.

[0115] The information received by the view generation module is processed by the view extraction layer, which effectively improves the operation efficiency of the view generation module.

[0116] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective management layer may include at least one perspective dimension information;

[0117] Specifically, the initial viewing angle feature information is input into the viewing angle management layer to generate reference viewing angle feature information, including:

[0118] Reference viewing angle feature information is generated according to the initial viewing angle feature information and each viewing angle dimension information.

[0119] In actual applications, the viewing angle management layer receives initial viewing angle feature information and can generate reference viewing angle feature information based on information of each dimension.

[0120] Specifically, each dimension information in the perspective management layer is initialized based on each sample image initially input at the beginning of training, and will be adjusted according to the input sample image during the training process, wherein the number of dimensional perspective information is any positive integer. Furthermore, the role of the perspective management layer can be understood as managing the perspective dimension through the input sample image, so that the perspective dimension information can be more closely matched with the actual application scenario, thereby improving the accuracy of perspective dimension processing. The number of dimensions can be understood as a preliminary classification of sample images according to the dimensions, and the sample images are processed according to the dimensional information after classification to fuse image features of different dimensions.

[0121] In one embodiment provided in the present specification, the number of dimensions is set to 64. At the beginning of training, 64 initial perspective feature information initially input to the perspective management layer are initialized as 64 perspective dimension parameters, the initial perspective feature information subsequently input is classified according to the above 64 perspective dimensions, the perspective dimension parameters are adjusted based on the classification of the perspective dimensions, and the parameters of the 64 perspective dimensions are integrated into the initial perspective feature information to generate reference perspective feature information.

[0122] By performing multi-dimensional processing on the initial perspective feature information based on the perspective dimension parameters after the perspective management operation, images of multiple perspectives can be fully utilized for training, thereby improving the accuracy of the image processing model when processing images.

[0123] In order to fully utilize images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images, in one or more embodiments of this specification, the perspective transformation layer may include at least one perspective transformation matrix and at least one perspective transformation parameter;

[0124] Furthermore, the reference viewing angle feature information is input into the viewing angle transformation layer to generate target viewing angle feature information, including:

[0125] The target viewing angle feature information is obtained according to the reference viewing angle feature information and each viewing angle transformation matrix and each viewing angle transformation parameter, wherein each viewing angle transformation matrix and each viewing angle transformation parameter are generated by each viewing angle dimension information.

[0126] In practical applications, a view transformation matrix and view transformation parameters are generated based on the dimensional information included in the reference view feature information, and target view feature information is generated based on the view transformation matrix and the transformation parameters.

[0127] Specifically, the perspective transformation matrix is ​​generated based on the dimensional information in the reference perspective feature information, and the perspective transformation parameters are also generated based on the dimensional information in the reference perspective feature information. Further, the perspective transformation matrix and the perspective transformation parameters can be understood as processing the image feature information based on multiple perspective dimensions so that the image feature information contains more perspective feature information, and thus the image of a single perspective can contain information of more dimensional perspectives after processing.

[0128] By processing image feature information based on multiple perspective dimensions, the image feature information contains more perspective feature information, and thus an image from a single perspective can contain information from more dimensions of perspective after processing, so that images from multiple perspectives can be fully utilized for training, thereby improving the accuracy of the image processing model when processing images.

[0129] Specifically, inputting the first-view image feature information and the second-view image feature information into the basic module to obtain a predicted image processing result includes:

[0130] Inputting the first-view image feature information and the second-view image feature information into a second coding layer to obtain mixed-view image feature information;

[0131] Inputting the mixed-view image feature information into a target coding layer set to obtain target coding image feature information, wherein the target coding layer set includes an integer number of coding layers;

[0132] The target encoded image feature information is input into the decoder to obtain a predicted image processing result.

[0133] In practical applications, the second-perspective image feature information and the first-perspective image feature information generated by the perspective generation module can be input into the second coding layer to obtain mixed perspective feature information, and then the mixed perspective feature information can be input into the target coding layer set to obtain target coded image feature information, and the obtained target coded image feature information can be input into the decoder to obtain the image processing results required for the image processing task. The image processing results specifically include images with specific target identifications, images with classification identifications, and so on.

[0134] Specifically, the second coding layer is a coding layer that inputs the first-perspective image feature information and the second-perspective image feature information, and does not specifically refer to a coding layer at a certain position. The target coding layer set is a coding layer set including an integer number of coding layers. Further, the second coding layer can be understood as combining the first-perspective image feature information and the second-perspective image feature information to generate a mixed perspective feature information that combines perspective features of multiple dimensions. The target coding layer set can be understood as setting multiple coding layers to deepen the application of the image processing model to the obtained mixed perspective feature information.

[0135] In one embodiment provided in the present specification, the target coding layer set is set to 2 layers and the model of the basic module is the SegFormer model, then the first perspective information is set to the Q tensor, the second perspective information is set to the K tensor and the V tensor, the three tensors Q, K, and V are input into the second coding layer to obtain mixed perspective image feature information, and then the mixed perspective image feature information is input into the first coding layer in the target coding layer set to obtain an intermediate image feature matrix, and then the intermediate image feature matrix is ​​input into the next coding layer in the target coding layer set to obtain the target coding image feature information. Since the model of the basic module is the SegFormer model, the decoder is configured as a SegFormer model decoder, and the target coding image feature information is input into the SegFormer model decoder to obtain an image with a specific target identifier corresponding to the sample image.

[0136] The first-perspective information and the second-perspective information enhance the image processing model's ability to acquire information about images from multiple perspectives, thereby making full use of images from multiple perspectives for training, thereby improving the accuracy of the image processing model when processing images.

[0137] Step 206: Calculate a model loss value according to the predicted image processing result and the sample image processing result.

[0138] After obtaining the processing results of the image processing model, the model loss value can be calculated based on the image processing results and the sample processing results. In the method provided in this specification, there are many methods for calculating the model loss value, such as cross entropy loss function, maximum loss function, average loss function, etc. In this specification, the specific method of the loss function is not limited, and it is subject to actual application.

[0139] Step 208: Adjust the model parameters of the image processing model according to the model loss value, and continue to train the image processing model until the model training stop condition is reached.

[0140] After obtaining the model loss value, the model parameters of the image processing model can be adjusted according to the model loss value. Specifically, the model loss value can be back-propagated to update the model parameters of the image processing model in sequence.

[0141] After adjusting the model parameters, the above steps can be repeated to continue training the image processing model until the training stop condition is reached. In practical applications, the training stop condition of the image processing model includes:

[0142] The model loss value is less than a preset threshold; and / or

[0143] The training rounds reach the preset training rounds.

[0144] Specifically, in the process of training the image processing model, the training stop condition of the model can be set to a model loss value less than a preset threshold, or the training stop condition can be set to a preset number of training rounds, such as 10 rounds of training. In this specification, there is no specific limitation on the preset threshold of the loss value and / or the preset number of training rounds, which shall be subject to actual application.

[0145] In addition, after the model training stop condition is reached, the image processing model training method further includes:

[0146] The viewing angle generation module is removed from the image processing model.

[0147] In practical applications, after the image processing model is trained, only the basic module can be retained to perform model reasoning operations.

[0148] Specifically, the perspective generation module is a module that performs a perspective generation operation on the feature information of the first perspective image to generate feature information of the second perspective image. Furthermore, the perspective generation module can be understood as the operation in the perspective generation module can enhance the ability of the image processing model to establish connections between images of multiple perspectives, and the perspective generation module does not contain any neural network structure, and then after the model training is completed, only the trained basic module can be used to perform the image processing process.

[0149] By using the perspective generation module when training the model, the image processing model's ability to establish connections between images from multiple perspectives is enhanced, thereby achieving greater accuracy in processing images from multiple perspectives. In addition, when performing image processing tasks, the perspective generation module is eliminated and only the basic modules trained based on the perspective generation module are used, further increasing the efficiency of image processing using the above-mentioned image processing model.

[0150] In addition, after the model training stop condition is reached, the image processing model training method further includes:

[0151] Obtain model parameters of the image processing model, and send the model parameters of the image processing model to the end-side device.

[0152] In practical applications, the sample images and sample image processing results provided by the client can be received to train the above image processing model in the cloud, and the image processing model parameters trained based on the sample images and sample image processing results uploaded by the client can be sent to the client.

[0153] By sending image processing model parameters to the client, a cloud-based model training service can be provided to apply the client's model training method on the cloud, thereby saving the resources required by the client.

[0154] By applying the scheme of the above embodiment, the sample image is input into the embedding layer and the encoder set to generate the first-perspective image feature information. The generated first-perspective image feature information has the initial features of the sample image, which is convenient for the subsequent processing of the sample image; the first-perspective image feature information is then input into the perspective generation module to obtain the second-perspective image feature information. The second-perspective image feature information generated by the perspective generation module can be understood as the fusion of the perspective feature information of multiple dimensions into the second-perspective image feature information according to the original first-perspective image feature information of the sample image, so as to facilitate the subsequent processing of the sample image; then the first-perspective information and the second-perspective information are combined for image prediction. The purpose of this step is to combine the original first-perspective image feature information of the sample image with the second-perspective image feature information with the perspective features of multiple dimensions, so as to achieve the fusion of the sample image with the characteristic information of multiple perspective dimensions, and then when the image processing model is trained, the trained model can link the perspective information of multiple dimensions; thereby, the image processing model can make full use of the images of multiple perspectives for training, and further, the image processing model trained by the image processing model training method can improve the accuracy of image processing when processing images.

[0155] The following combination Figure 4 , taking the application of the image processing method provided in this specification in the recognition of target objects as an example, the recognition method of the target object is further described. Figure 4 A process flow chart of a target object recognition method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0156] Step 402: Obtain the image to be processed collected by the drone.

[0157] In practical applications, a video segment containing multiple images captured by a drone can be obtained, and key frames can be extracted as images to be processed.

[0158] Specifically, the image to be processed is a key frame of a video captured by a drone. Furthermore, the image to be processed can be any color image or grayscale image, a video clip composed of multiple images, and the like.

[0159] In one embodiment provided in this specification, a video segment that is not collected by an unmanned person is received and a key frame in the segment is obtained, and the key frame is regarded as an image to be processed.

[0160] By acquiring the unprocessed images transmitted by the drone, subsequent image processing is facilitated.

[0161] Step 404: input the image to be processed into an image processing model to obtain a water surface area and a target object area in the image to be processed output by the image processing model, wherein the image processing model is obtained by training using the image processing model training method provided in the embodiments of this specification.

[0162] In practical applications, the image to be processed can be input into the image processing model trained in the above-mentioned image processing model training method to obtain the water surface area and the target object area in the image to be processed.

[0163] Specifically, the water surface area is all image areas in the image marked as water surfaces by the image processing model, and the target object area is all image areas in the image marked as target objects by the image processing model, where the target objects can be humans, buildings, vehicles, etc.

[0164] In one embodiment provided in this specification, the image to be processed is input into the above-mentioned image processing model, an image with marked water surface area and image area is obtained, and the water surface area and image area in the image are extracted to facilitate subsequent operations.

[0165] The water surface area and the image area in the image are extracted to facilitate the subsequent processing of the water surface area and the target object area.

[0166] Step 406: Generate a shore area according to the water surface area.

[0167] In practical applications, the shore area can be generated based on the water surface area processing.

[0168] Specifically, the shore area is a corresponding area generated according to the water surface area. Furthermore, the shore area can be understood as the water surface area expanded and then the original water surface area subtracted.

[0169] The shore area is obtained through the water surface area, so as to subsequently obtain the target object contained in the shore area.

[0170] Specifically, generating a shore area according to the water surface area includes:

[0171] Determining an initial water surface area based on a preset water surface threshold and the water surface area;

[0172] Based on the initial water surface area, a shore area is generated.

[0173] In practical applications, a water surface area that meets the conditions can be selected according to a preset water surface threshold, and then the shore area can be determined according to the water surface area that meets the conditions.

[0174] Specifically, the water surface threshold is a value preset by those skilled in the art when using the method. Further, the preset water surface threshold can be understood as screening out water surfaces that meet the water surface area size. For example, if the threshold is set to 10, then water surfaces with an area of ​​not less than 10m2 will be screened out in the image. 2 The water surface is taken as the initial water surface area

[0175] In one embodiment provided in this specification, the preset water surface threshold is 5, and the image processing results above 5m are selected. 2 The water surface is taken as the initial water surface area, and the shore area is generated according to the selected initial water surface area.

[0176] By screening out the water surface areas that meet the target range, errors in the target objects obtained in the final result can be avoided.

[0177] Furthermore, based on the initial water surface area, generating a shore area includes:

[0178] Performing polygonal approximation processing on the initial water surface area to generate a reference water surface area;

[0179] Performing polygon expansion processing on the reference water surface area to generate a target water surface area;

[0180] The shore area is generated according to the difference between the reference water surface area and the target water surface area.

[0181] In practical applications, the initial water surface area can be processed by polygonal approximation to generate a reference water surface area, and then the reference water surface area can be processed by polygonal expansion to generate a target water surface area, and finally the shore area can be generated based on the difference between the reference water surface area and the target water surface area.

[0182] Specifically, polygonal approximation is to represent the target area with polygons with fewer vertices, and polygonal expansion is to update the vertices of the polygons according to the polygonal expansion algorithm. Furthermore, the expansion after polygonal approximation of the original water surface area can be understood as the expansion of the initial water surface area.

[0183] It should be noted that the existing method of expanding the water surface area usually involves first binarizing the image to obtain the target to be expanded. Morphological expansion is to expand the boundary points of the binary object, merge all background points that are in contact with the object into the object, and expand the boundary outward. If the two objects are close to each other, the two objects will be connected together, and different sizes of expanded object boundaries will be obtained according to different expansion distances. The calculation steps required by this method are cumbersome and the calculation pressure on the computer is high. The polygon approximation and polygon expansion algorithm provided in this manual greatly reduce the calculation steps.

[0184] In one embodiment provided in this specification, an initial water surface area represented by 100 coordinate points is approximated and represented by 10 vertices, and then a polygon expansion algorithm is used for the approximated reference water surface area, as shown in Formula Group 1:

[0185]

[0186] Where P i is the vertex after expansion, Q i is the original vertex, L is the expansion (scaling) distance, and M is the vector The size of and is a vector The component vectors of and The angle between the two, Normalize(v1) and Normalize(v2) respectively represent and The unit vector of .

[0187] The shore area is obtained by subtracting the target water surface area after expansion from the initial water surface area.

[0188] By using polygons with fewer vertices to represent the initial water surface and approximating the water surface area using polygon expansion, the computing resources required to expand the water surface area are greatly reduced. The shore area is confirmed by the difference between the water surface area after expansion and the initial water surface area, which improves the efficiency of calculating the shore area based on the water surface area, thereby improving the efficiency of confirming the target object.

[0189] Step 408: Determine the recognition result of the target object according to the comparison result between the target object area and the shore area.

[0190] In practical applications, the target object area and the shore area can be compared, and the recognition result of the target object can be determined based on the comparison result.

[0191] Specifically, the target object recognition result is to mark out the target objects that meet the requirements, such as marking out the buildings on the shore, marking out the cars parked on the shore, and so on.

[0192] In one embodiment provided in this specification, a cross-validation calculation is performed between the target object area and the shore area to obtain the target object on the shore and mark it in the image.

[0193] By inputting the image to be processed acquired by the drone into the above-mentioned image processing model, the recognition result of the target object is obtained according to the image processing result output by the image processing model, thereby achieving accurate automatic recognition of the target object.

[0194] The scheme of the above embodiment is applied, and the image collected by the drone is input into the image processing model trained according to the image processing model training method provided by the embodiment of this specification to obtain the images of the marked water surface area and the target object area. The above image processing model is applied to ensure the accuracy of processing the image collected by the drone; then the water surface area to be expanded is selected according to the preset threshold, which effectively avoids the expansion of the water surface area that does not need to be processed, and reduces the redundant performance overhead when expanding the water surface area; when expanding the water surface area, the water surface area is firstly processed by polygon approximation, and then the polygon expansion processing is performed according to the approximated polygon. Since the water surface area is processed by multivariate approximation, the amount of calculation when the polygon is expanded is greatly reduced. At the same time, the polygon expansion algorithm used when the polygon is expanded greatly improves the efficiency of the operation compared with the processing of the water surface area based on the morphological expansion algorithm; the water surface area after expansion is subtracted from the original water surface area to obtain the shore area, and the required target object is obtained based on the relationship between the obtained shore area and the target object area, thereby realizing automatic detection of the target object, and then the drone can quickly and automatically identify the target object.

[0195] The following combination Figure 5 Taking the application of the image processing model training method provided in this specification in the semantic segmentation model training as an example, the semantic segmentation model training method is further described. Figure 5 A flowchart of a processing process of a semantic segmentation model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0196] Step 502: Obtain a sample image and a segmentation result corresponding to the sample image.

[0197] Step 504: Input the sample image into the embedding layer in the image processing model to obtain sample image encoding information.

[0198] Step 506: Input the sample image coding information into the first coding layer in the basic module to obtain the first-view image feature information, wherein the basic module contains 4 coding layers.

[0199] Step 508: input the first-view image feature information into the view extraction layer in the view generation module, and obtain the initial view feature information through the view encoding unit in the view extraction layer.

[0200] Step 510: Input the initial viewing angle feature information into the viewing angle management layer to obtain reference viewing angle feature information.

[0201] Step 512: Input the reference viewing angle feature information into the viewing angle transformation layer to obtain the target viewing angle feature information.

[0202] Step 514: input the target viewing angle feature information into the viewing angle extraction layer, and obtain the second viewing angle image feature information through the viewing angle decoding unit of the viewing angle extraction layer.

[0203] Step 516: input the first-view image feature information and the second-view image feature information into the second coding layer to obtain the first image feature information.

[0204] Step 518: Input the first image feature information into the third coding layer to obtain the second image feature information.

[0205] Step 520: Input the second image feature information into the fourth coding layer to obtain the third image feature information.

[0206] Step 522: input the first-view image feature information, the first image feature information, the second image feature information and the third image feature information into a decoding layer in a decoder to obtain sample image feature information.

[0207] Step 524: Input the sample image feature information into the image segmentation layer in the decoder to obtain the predicted sample image segmentation result.

[0208] Step 526: Calculate the loss value of the model prediction result by combining the predicted sample image segmentation result and the segmentation result corresponding to the sample image.

[0209] Step 528: Adjust the parameters of the image processing model according to the obtained prediction loss value.

[0210] The scheme of the above embodiment is applied, by inputting the sample image into the embedding layer and the encoder set to generate the first-perspective image feature information, the generated first-perspective image feature information has the initial features of the sample image, which is convenient for the subsequent processing of the sample image; then the first-perspective image feature information is input into the perspective generation module to obtain the second-perspective image feature information, the second-perspective image feature information generated by the perspective generation module can be understood as the fusion of the perspective feature information of multiple dimensions into the second-perspective image feature information according to the original first-perspective image feature information of the sample image, so as to facilitate the subsequent processing of the sample image; then the first-perspective information and the second-perspective information are combined for image prediction, the purpose of this step is to combine the original first-perspective image feature information of the sample image with the second-perspective image feature information with the perspective features of multiple dimensions, so as to achieve the fusion of the sample image with the characteristic information of multiple perspective dimensions, and then when the image processing model is trained, the trained model can link the perspective information of multiple dimensions; thereby, the image processing model can make full use of the images of multiple perspectives for training, and further, the image processing model trained by the image processing model training method can improve the accuracy of processing images.

[0211] With the rapid development of drone technology, drones have been widely used in research and industrial applications. Images and videos are important ways for drones to perceive their surroundings. Drones are often in motion during flight, which causes the image perspectives collected by drones to change constantly, making the accuracy of existing image processing methods applied to drones lower than expected. Based on this, one or more specific implementations provided in this specification include an image processing method. Specifically, the following combined with the attached Figure 6 , taking the application of the image processing method provided in this specification in the recognition of target objects as an example, the image processing method is further described. Figure 6 A process flow chart of a target object recognition method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0212] Step 602: receiving an image processing request sent by a user, wherein the image processing request carries an image to be processed.

[0213] In actual applications, an image processing request sent by a user may be received, and the image to be processed carried in the request may be obtained.

[0214] Specifically, the image processing request includes the image processing task type and the image to be processed. The image to be processed can be any color image or grayscale image, a video clip composed of multiple images, etc. The task type can be a semantic segmentation task, an image classification task, etc.

[0215] In one embodiment provided in this specification, a user inputs a video clip shot by a drone, and needs to obtain a feature image containing possible ruins, relics, and geographical features in the video clip. It can be determined that the task type required by the user is an image classification task, and the image processing result to be output is a feature image classified as containing possible ruins, relics, and geographical features. The video clip input by the user is subjected to key extraction processing to obtain the image to be processed.

[0216] The image to be processed in the request is obtained so as to use the image processing model described below to process the obtained image to be processed.

[0217] Step 604: input the image to be processed into an image processing model to obtain an image processing result output by the image processing model, wherein the image processing model is obtained by training using the image processing model training method provided in the embodiments of this specification.

[0218] In practical applications, the acquired image to be processed can be input into an image processing model trained by the image processing model training method provided by the embodiments of this specification to obtain an image processing result.

[0219] In one embodiment provided in this specification, key frame images extracted from video clips are input into the image processing model completed for training to obtain the classification of each key frame image.

[0220] By inputting the image to be processed into the image processing model trained by the image processing model training method provided in the embodiments of this specification, the image processing result can be obtained to perform high-precision processing on images containing multiple perspectives.

[0221] Step 606: Send the image processing result to the user.

[0222] In practical applications, the image processing results can be sent to users.

[0223] In one embodiment provided in the present specification, feature images classified as containing possible ruins, relics and geographical features are selected and sent to users based on the classification results output by the above-mentioned image processing model.

[0224] By processing the image processing task input by the user using an image processing model trained based on the image processing model training method provided in the embodiments of this specification, high-precision processing of images containing multiple perspectives is achieved.

[0225] By applying the solution of the above-mentioned embodiment, it is possible to accept image processing requests sent by users, and to process the received image processing requests using the image processing model trained by the image processing model training method provided by the embodiments in this specification, and to return the processing results to the user. Since the image processing model trained by the image processing model training method provided by the embodiments in this specification can accurately process images, the image processing method can accurately process image processing requests sent by users.

[0226] Corresponding to the above method embodiment, this specification also provides an image processing model training device embodiment, Figure 7 FIG. 1 is a schematic diagram showing the structure of an image processing model training device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0227] An acquisition module 702 is used to acquire a sample image and a sample image processing result corresponding to the sample image;

[0228] The prediction module 704 inputs the sample image into the image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated according to the first-view image feature information and the second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information;

[0229] A judgment module 706 calculates a model loss value according to the predicted image processing result and the sample image processing result;

[0230] The stop module 708 adjusts the model parameters of the image processing model according to the model loss value, and continues to train the image processing model until the model training stop condition is reached.

[0231] Optionally, the image processing model includes a basic module and a viewing angle generation module;

[0232] The prediction module 704 is further configured to:

[0233] Inputting the sample image into the basic module to obtain first-view image feature information;

[0234] Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information;

[0235] The first-view image feature information and the second-view image feature information are input into the basic module to obtain a predicted image processing result.

[0236] Optionally, the basic module includes an encoder and a decoder;

[0237] The prediction module 704 is further configured to:

[0238] The sample image is input into the encoder to obtain first-view image feature information.

[0239] Optionally, the encoder comprises an embedding layer and at least two encoding layers;

[0240] The prediction module 704 is further configured to:

[0241] Inputting the sample image into the embedding layer to obtain sample image encoding information;

[0242] Inputting the sample image coding information into an initial coding layer set to obtain initial coding image feature information, wherein the initial coding layer set includes an integer number of coding layers;

[0243] The initial encoded image feature information is input into the first encoding layer to obtain the first viewing angle image feature information.

[0244] Optionally, the perspective generation module includes a perspective extraction layer, a perspective management layer and a perspective transformation layer;

[0245] The prediction module 704 is further configured to:

[0246] Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information;

[0247] Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information;

[0248] Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information;

[0249] The target viewing angle feature information is input into the viewing angle extraction layer to generate second viewing angle image feature information.

[0250] Optionally, the view extraction layer includes a view encoding unit and a view decoding unit;

[0251] The prediction module 704 is further configured to:

[0252] Inputting the first viewing angle image information into the viewing angle encoding unit to generate initial viewing angle feature information;

[0253] Accordingly, the target viewing angle feature information is input into the viewing angle extraction layer to generate the second viewing angle image feature information, including:

[0254] The target viewing angle feature information is input into the viewing angle decoding unit to generate second viewing angle image feature information.

[0255] Optionally, the viewing angle management layer includes at least one viewing angle dimension information;

[0256] The prediction module 704 is further configured to:

[0257] Reference viewing angle feature information is generated according to the initial viewing angle feature information and each viewing angle dimension information.

[0258] Optionally, the perspective transformation layer includes at least one perspective transformation matrix and at least one perspective transformation parameter, wherein each perspective transformation matrix and each perspective transformation parameter are generated by each perspective dimension information;

[0259] The prediction module 704 is further configured to:

[0260] The target viewing angle feature information is obtained according to the reference viewing angle feature information, each viewing angle transformation matrix and each viewing angle transformation parameter.

[0261] Optionally, the prediction module 704 is further configured to:

[0262] Inputting the first-view image feature information and the second-view image feature information into a second coding layer to obtain mixed-view image feature information;

[0263] Inputting the mixed-view image feature information into a target coding layer set to obtain target coding image feature information, wherein the target coding layer set includes an integer number of coding layers;

[0264] The target encoded image feature information is input into the decoder to obtain a predicted image processing result.

[0265] Optionally, the image processing model training device further includes:

[0266] A removal module removes the viewing angle generation module from the image processing model after the model training stop condition is reached.

[0267] Optionally, the image processing model training device further includes:

[0268] The sending module obtains the model parameters of the image processing model after the model training stop condition is reached, and sends the model parameters of the image processing model to the terminal side device.

[0269] The above is a schematic scheme of an image processing model training device of this embodiment. It should be noted that the technical scheme of the image processing model training device and the technical scheme of the above-mentioned image processing model training method belong to the same concept, and the details not described in detail in the technical scheme of the image processing model training device can be found in the description of the technical scheme of the above-mentioned image processing model training method.

[0270] Figure 8 The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0271] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0272] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0273] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0274] Among them, the processor 820 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0275] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned image processing model training method or image processing method or target object recognition method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0276] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0277] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned image processing model training method or image processing method or target object recognition method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0278] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0279] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned image processing model training method or image processing method or target object recognition method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned image processing model training method or image processing method or target object recognition method.

[0280] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0281] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0282] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0283] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0284] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for training an image processing model, comprising: Acquire a sample image and a sample image processing result corresponding to the sample image; Input the sample image into an image processing model to obtain a predicted image processing result output by the image processing model, wherein the predicted image processing result is generated according to first-view image feature information and second-view image feature information, the first-view image feature information is generated based on the sample image, and the second-view image feature information is generated based on the first-view image feature information; Calculate the model loss value according to the predicted image processing result and the sample image processing result; The model parameters of the image processing model are adjusted according to the model loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

2. The method according to claim 1, wherein the image processing model comprises a basic module and a perspective generation module; Inputting the sample image into an image processing model to obtain a predicted image processing result output by the image processing model includes: Inputting the sample image into the basic module to obtain first-view image feature information; Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information; The first-view image feature information and the second-view image feature information are input into the basic module to obtain a predicted image processing result.

3. The method of claim 2, wherein the basic module comprises an encoder and a decoder, wherein the encoder comprises an embedding layer and at least two encoding layers; Inputting the sample image into the basic module to obtain first-view image feature information includes: Inputting the sample image into the embedding layer to obtain sample image encoding information; Inputting the sample image coding information into an initial coding layer set to obtain initial coding image feature information, wherein the initial coding layer set includes an integer number of coding layers; The initial encoded image feature information is input into the first encoding layer to obtain the first viewing angle image feature information.

4. The method according to claim 2, wherein the view generation module comprises a view extraction layer, a view management layer and a view transformation layer; Inputting the first-view image feature information into the view generation module to obtain the second-view image feature information includes: Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information; Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information; Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information; The target viewing angle feature information is input into the viewing angle extraction layer to generate second viewing angle image feature information.

5. The method of claim 4, wherein the view extraction layer comprises a view encoding unit and a view decoding unit; Inputting the first viewing angle image information into the viewing angle extraction layer to generate initial viewing angle feature information includes: Inputting the first viewing angle image information into the viewing angle encoding unit to generate initial viewing angle feature information; Accordingly, the target viewing angle feature information is input into the viewing angle extraction layer to generate the second viewing angle image feature information, including: The target viewing angle feature information is input into the viewing angle decoding unit to generate second viewing angle image feature information.

6. The method according to claim 4, wherein the viewing angle management layer comprises at least one viewing angle dimension information; Inputting the initial viewing angle feature information into the viewing angle management layer to generate reference viewing angle feature information includes: Reference viewing angle feature information is generated according to the initial viewing angle feature information and each viewing angle dimension information.

7. The method of claim 6, wherein the perspective transformation layer comprises at least one perspective transformation matrix and at least one perspective transformation parameter, wherein: Each viewing angle transformation matrix and each viewing angle transformation parameter are generated from each viewing angle dimension information; Inputting the reference viewing angle feature information into the viewing angle transformation layer to generate target viewing angle feature information includes: The target viewing angle feature information is obtained according to the reference viewing angle feature information, each viewing angle transformation matrix and each viewing angle transformation parameter.

8. The method according to claim 3, inputting the first-view image feature information and the second-view image feature information into the basic module to obtain a predicted image processing result, comprising: Inputting the first-view image feature information and the second-view image feature information into a second coding layer to obtain mixed-view image feature information; Inputting the mixed-view image feature information into a target coding layer set to obtain target coding image feature information, wherein the target coding layer set includes an integer number of coding layers; The target encoded image feature information is input into the decoder to obtain a predicted image processing result.

9. An image processing method, comprising: Receiving an image processing request sent by a user, wherein the image processing request carries an image to be processed; Inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model, wherein the image processing model is obtained by the method according to any one of claims 1 to 8; The image processing result is sent to the user.

10. A method for identifying a target object, comprising: Obtain the images to be processed collected by the drone; Inputting the image to be processed into an image processing model to obtain a water surface area and a target object area in the image to be processed output by the image processing model, wherein the image processing model is obtained by the method according to any one of claims 1 to 8; generating a shore area according to the water surface area; The recognition result of the target object is determined according to the comparison result between the target object area and the shore area.

11. The method according to claim 10, generating a shore area according to the water surface area, comprising: Determining an initial water surface area based on a preset water surface threshold and the water surface area; Based on the initial water surface area, a shore area is generated.

12. The method according to claim 11, generating a shore area based on the initial water surface area, comprising: Performing polygonal approximation processing on the initial water surface area to generate a reference water surface area; Performing polygon expansion processing on the reference water surface area to generate a target water surface area; The shore area is generated according to the difference between the reference water surface area and the target water surface area.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the image processing model training method described in any one of claims 1 to 8, the image processing method described in claim 9, or the target object recognition method described in any one of claims 10 to 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the image processing model training method described in any one of claims 1 to 8, the image processing method described in claim 9, or the target object recognition method described in any one of claims 10 to 12.

Images