Image processing method, storage medium and computer terminal

Abstract

This application discloses an image processing method, storage medium, and computer terminal. The method comprises: acquiring a target image, wherein the target image contains a target object; performing feature extraction on the target image to obtain an initial feature vector; performing semantic processing on the initial feature vector to obtain a semantic feature vector; and performing object detection on the target image based on the initial feature vector and the semantic feature vector to obtain an object detection result for the target object. This application addresses the technical problem in related arts of low accuracy of object detection results in target images due to insufficient feature expression of the target object.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, a storage medium, and a computer terminal. Background Art

[0002] Currently, ReDet (rotation equivariant detector) uses the information of the rotation box as a supervisory signal when solving the task of rotating target detection. The feature expression ability of the features obtained is limited, which limits the performance of ReDet. Therefore, the target detection results obtained by using ReDet to detect target objects in the target image are less accurate.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, a storage medium, and a computer terminal to at least solve the technical problem in the related art that the detection results of target detection on the target image have low accuracy due to insufficient feature expression of the target object.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, including: acquiring a target image, wherein the target image contains a target object; performing feature extraction on the target image to obtain an initial feature vector; performing semantic processing on the initial feature vector to obtain a semantic feature vector; and performing target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0006] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: obtaining a building image, wherein the building image contains a building; performing feature extraction on the building image to obtain an initial feature vector; performing semantic processing on the initial feature vector to obtain a semantic feature vector; and performing building detection on the building image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the building.

[0007] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: a cloud server receives a target image uploaded by a client, wherein the target image contains a target object; the cloud server performs feature extraction on the target image to obtain an initial feature vector; the cloud server performs semantic processing on the initial feature vector to obtain a semantic feature vector; the cloud server performs target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object; the cloud server sends the target detection result to the client.

[0008] According to one aspect of an embodiment of the present application, an image processing device is provided, including: an acquisition module for acquiring a target image, wherein the target image contains a target object; an extraction module for performing feature extraction on the target image to obtain an initial feature vector; a processing module for performing semantic processing on the initial feature vector to obtain a semantic feature vector; and a detection module for performing target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0009] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: an acquisition module for acquiring a building image, wherein the building image contains a building; an extraction module for performing feature extraction on the building image to obtain an initial feature vector; a processing module for performing semantic processing on the initial feature vector to obtain a semantic feature vector; and a detection module for performing building detection on the building image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the building.

[0010] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a receiving module for receiving a target image uploaded by a client through a cloud server, wherein the target image includes a target object; an extraction module for performing feature extraction on the target image through the cloud server to obtain an initial feature vector; a processing module for performing semantic processing on the initial feature vector through the cloud server to obtain a semantic feature vector; a detection module for performing target detection on the target image based on the initial feature vector and the semantic feature vector through the cloud server to obtain a target detection result of the target object; and a sending module for sending the target detection result to the client through the cloud server.

[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned image processing method.

[0012] According to another aspect of an embodiment of the present application, a computer terminal is further provided, including a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the above-mentioned image processing method is executed when the program is run.

[0013] In an embodiment of the present application, first, a target image containing a target object can be obtained, and feature extraction can be performed on the target image to obtain an initial feature vector, and then semantic processing can be performed on the initial feature vector to obtain a semantic feature vector; based on the initial feature vector and the semantic feature vector, target detection can be performed on the target image to obtain a target detection result of the target object, thereby achieving the effect of improving the accuracy of the detection result. It is easy to notice that by performing semantic processing on the initial feature vector to obtain a semantic feature vector, the rich semantic feature vector can be fused and complemented with the initial feature vector, thereby improving the feature expression capability of selecting the target object, thereby improving the accuracy of target detection on the target image, thereby solving the technical problem in the related art that the detection result accuracy of target detection on the target image is low due to insufficient feature expression capability of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method according to the prior art;

[0016] Figure 2 is a flowchart of an image processing method according to Example 1 of the present application;

[0017] Figure 3 is a flowchart of another image processing method according to Example 1 of the present application;

[0018] Figure 4 The image is obtained by marking the target object in the remote sensing image according to Example 1 of the present application;

[0019] Figure 5 is a schematic diagram of an image processing process according to Example 1 of the present application;

[0020] Figure 6 is a flowchart of an image processing method according to Example 2 of the present application;

[0021] Figure 7 is a flowchart of an image processing method according to Example 3 of the present application;

[0022] Figure 8 is a schematic diagram of an image processing device according to Embodiment 4 of the present application;

[0023] Figure 9 is a schematic diagram of an image processing device according to Embodiment 5 of the present application;

[0024] Figure 10 is a schematic diagram of an image processing device according to Example 6 of the present application;

[0025] Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0029] RPN (Region Proposal Network): extract candidate boxes;

[0030] RoI (Region of interest): region of interest;

[0031] RT (Region of interest Transform): rotate the region of interest;

[0032] Currently, ReDet is a relatively advanced two-step rotation detection framework. It rewrites ResNet50 using a rotation equivariant operator, allowing the backbone network to learn rotation-equivariant features to a certain extent. However, ReDet only uses the rotated target box as a supervisory signal for training, so the learned features are limited in their expressiveness, which in turn limits their performance.

[0033] In order to solve the above problems, the present application provides an image processing method, which performs multi-task learning through semantic feature auxiliary branches, which can further enhance the feature expression of rotating objects, thereby further improving the detection performance of rotating targets.

[0034] Example 1

[0035] According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be fully or partially integrated into any of the other components of the computer terminal 10 (or mobile device). The data processing circuitry may function as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned image processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the aforementioned computer device (or mobile device).

[0042] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0043] Step S202: Acquire a target image.

[0044] The target image contains the target object.

[0045] The target image in the above steps can be a remote sensing image, an aerial image taken by a drone, or a radar image, but is not limited to these. In different application scenarios, the types of target objects contained in the target image vary. For example, in urban planning applications, the target image may contain road networks, land parcels, buildings, etc.; in transportation applications, the target image may contain vehicles; in agriculture and forestry applications, the target image may contain land parcels, greenhouses, etc.; in meteorological applications, the target image may contain clouds; in water conservancy applications, the target image may contain dams, rivers, lakes, etc.; and in disaster applications, the target image may contain mountains, dams, etc.

[0046] In an optional embodiment, the target image can be captured by a satellite or drone and transmitted to a server via a network for processing by the server. The target image can also be displayed to the user and displayed within the image acquisition area. In another optional embodiment, the target image can be captured by a satellite or drone and uploaded to the server by the user and processed by the server. The user can upload the target image to the server by clicking the "Upload Image" button in the interactive interface or by directly dragging the target image into the dotted box. Moreover, the image uploaded by the user can be displayed within the image acquisition area. The server here can be a server deployed locally or in the cloud.

[0047] Step S204: extract features from the target image to obtain an initial feature vector.

[0048] In an optional embodiment, a feature extraction network may be used to extract features from the target image to obtain an initial feature vector.

[0049] The feature extraction network may be ResNet50+FPN (residual network and feature pyramid network). In order to improve the accuracy of the target object detection result in the target image, ResNet50+FPN may be used to extract features of the target image to obtain an initial feature vector.

[0050] Step S206: Perform semantic processing on the initial feature vector to obtain a semantic feature vector.

[0051] In an optional embodiment, the feature vectors of features such as contour, edge, color, texture and shape in the initial feature vector can be semantically processed to obtain a semantic feature vector. Through the semantic feature vector, differentiated features can be learned, thereby improving the expressive power of the features. The semantic feature vector and the initial feature vector can be combined to perform target detection on the target image to improve the accuracy of the detection results.

[0052] In another optional embodiment, a semantic segmentation model may be used to perform semantic processing on the initial feature vectors to obtain a semantic feature vector, wherein the semantic segmentation model is used to process the association relationship between the initial feature vectors based on the semantic information of the initial feature vectors to obtain the semantic feature vector.

[0053] In another optional embodiment, a semantic segmentation model can be used to semantically describe the initial feature vector in order to increase the expressive power of the feature. By obtaining the semantic feature vector, the number of feature vectors and feature vectors containing more information can be further increased. By combining the initial feature vector and the semantic feature vector to obtain a fused feature vector, the information contained in the feature can be enriched, thereby improving the accuracy of the target detection results.

[0054] Step S208 : performing target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0055] In an optional embodiment, after obtaining the semantic feature vector, the initial feature vector can be fused with the semantic feature vector to increase the number and richness of features, thereby improving the expressiveness of the features, and then performing target detection on the target image through the fused feature vector to improve the accuracy of the target detection results.

[0056] In another optional embodiment, the target detection result includes at least: the category of the target object, and the target detection frame corresponding to the target object, and the target detection frame is used to characterize the position of the target object in the target image. Among them, the target detection frame can be used to mark the position of the target object in the target image, wherein the target detection frame can frame the target object in the target image. In order to predict the target object in the target detection frame, the detection result of the target object is obtained by prediction. In another optional embodiment, the purpose of feature complementarity can be achieved based on the initial feature vector and the semantic feature vector. When performing target detection on the target image, the feature expression ability of the target object in the target image can be improved, so that the target object in the target image can be detected more accurately, so as to obtain a target detection result with higher accuracy.

[0057] In another optional embodiment, after performing target detection on the target image, the server can directly display the target detection result to the user for viewing, and the category of the target object displayed in the target detection result and the target detection box corresponding to the target object can be displayed in the target image in the result feedback area; in another optional embodiment, after performing target detection on the target image, the server can feed back the target detection result to the user's client through the network, and the client can display it to the user for viewing. Specifically, the client can display the category of the target object and the target detection box corresponding to the target object on the target image. If the user believes that the category of the target object and the position of the target detection box in the target detection result are incorrect, the user can adjust the position of the target detection box, re-enter the category of the target object, and upload it to the server, so that the server can re-train the model used for target detection based on the user's feedback, thereby improving the server performance.

[0058] In an urban planning scenario, the target image may be a building image, where the target objects contained in the building image may be buildings, bridges, sculptures, and other structures. A building image may be first acquired, and then features may be extracted from the building image using ResNet50+FPN to obtain an initial feature vector. The initial feature vector may then be semantically processed to obtain a semantic feature vector to better represent the features in the building image. The obtained initial feature vector and the semantic feature vector may then be fused, and the fused features may be used to perform target detection on the building image to obtain a target detection result for the building. The target detection result includes at least the building category and a target detection box corresponding to the building. The target detection box may be annotated at the location corresponding to the building in the target image.

[0059] Through the above steps, first, a target image containing a target object can be obtained, and features can be extracted from the target image to obtain an initial feature vector. Then, the initial feature vector is semantically processed to obtain a semantic feature vector. Based on the initial feature vector and the semantic feature vector, target detection is performed on the target image to obtain a target detection result of the target object, thereby achieving the effect of improving the accuracy of the detection result. It is easy to notice that by performing semantic processing on the initial feature vector to obtain a semantic feature vector, the rich semantic feature vector can be fused and complemented with the initial feature vector, thereby improving the feature expression capability of selecting the target object, thereby improving the accuracy of target detection on the target image, thereby solving the technical problem in the related art that the detection result accuracy of target detection on the target image is low due to insufficient feature expression capability of the target object.

[0060] In the above embodiment of the present application, the target detection result includes at least: a target detection frame, which is used to characterize the position of the target object in the target image, wherein target detection is performed on the target image based on the initial feature vector and the semantic feature vector, and the target detection result of the target object includes: determining the initial detection frame corresponding to the target object based on the initial feature vector; fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector; determining the target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain the target detection frame.

[0061] In an optional embodiment, the RPN can be used to identify the initial feature vector to obtain the initial horizontal detection frame of the target object in the initial feature vector, and then the initial horizontal detection frame is adjusted to an angled rotation frame according to RT, that is, the initial detection frame mentioned above.

[0062] Furthermore, the initial feature vector and the semantic feature vector can be fused to obtain a fused feature vector to improve the expressive ability of the feature. Since the fused feature vector can well characterize the characteristics of the target object, the target feature vector corresponding to the initial detection frame can be determined from the fused feature vector, thereby improving the accuracy of the target feature vector; after obtaining the target feature vector, the initial detection frame can be regressed according to the target feature vector, that is, the initial detection frame can be adjusted according to the target feature vector with higher accuracy, so as to obtain a target detection frame with higher accuracy. After obtaining the target detection frame, the feature vector in the target detection frame can be detected to determine the category of the target object in the target detection frame.

[0063] Furthermore, object recognition can be performed on the target feature vector so as to accurately obtain the category of the target object.

[0064] In the above embodiment of the present application, determining the initial detection frame corresponding to the target object based on the initial feature vector includes: processing the initial feature vector to obtain a detection frame of a preset angle corresponding to the target object; processing the detection frame of the preset angle to obtain the initial detection frame.

[0065] The aforementioned preset angle may be a pre-set angle, for example, an angle parallel to the x-axis of the geodetic coordinate system or an angle parallel to the y-axis of the geodetic coordinate system.

[0066] In an optional embodiment, the initial feature vector can be processed using RPN (Region Proposal Network) to obtain a detection frame of a preset angle corresponding to the target object, and then the detection frame of the preset angle can be adjusted using RT (Region of interest), that is, the detection frame of the preset angle is rotated so that the direction of the obtained initial detection frame is consistent with the direction of the target object, thereby improving the annotation accuracy of the initial detection frame.

[0067] In the above embodiment of the present application, the method also includes: obtaining training samples, wherein the training samples include: training images, and preset detection frames corresponding to the training objects contained in the training images; performing target detection on the training objects using a target detection model to obtain generated detection frames and target semantic segmentation results corresponding to the training objects, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module and a region conversion module, the feature extraction module is used to extract training feature vectors of the training images, the semantic segmentation module is used to generate target semantic segmentation results, the region generation module is used to generate training detection frames of preset angles, and the region conversion module is used to convert training detection frames of preset angles into generated detection frames; generating a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation results; and adjusting the model parameters of the target detection model based on the total loss function.

[0068] The above-mentioned feature extraction module may be ResNet50+FPN, the above-mentioned region generation module may be RPN, and the above-mentioned region conversion module may be RT.

[0069] In an optional embodiment, the target detection model can be trained by obtaining corresponding training samples for target images in different scenarios. For example, if the target detection model is to be used to process building images in an urban planning scenario, the parameters of the target detection model can be adjusted using training samples related to building images to achieve the purpose of training the target detection model.

[0070] The target semantic segmentation result may be the category of the training object and the preset detection box of the training object.

[0071] In another optional embodiment, the training objects in the training images can be detected based on a preset detection frame to determine the category of the training objects. The feature vectors corresponding to the target detection frame in the training feature vectors of the target semantic segmentation results are then detected to determine the category of the training objects. By comparing the categories of the training objects obtained in these two ways, the loss function between categories in the semantic segmentation results can be determined. By combining the loss function between detection frames and the loss function between categories to adjust the parameters of the target training model, the purpose of training the above modules can be achieved, further improving the accuracy of the target training model.

[0072] In the above embodiment of the present application, generating a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation result includes: obtaining the intersection-over-union ratio of the preset detection frame and the generated detection frame to generate a first loss function; obtaining the distance between the preset detection frame and the generated detection frame to generate a second loss function; generating a preset semantic segmentation result based on the preset detection frame; obtaining the distance between the preset semantic segmentation result and the target semantic segmentation result to generate a third loss function; generating a total loss function based on the first loss function, the second loss function and the third loss function.

[0073] In an optional embodiment, the intersection over union (IOU) of the preset detection frame and the generated detection frame can be obtained to measure the degree of overlap between the preset detection frame and the generated detection frame, so as to generate a first loss function based on the degree of overlap; the distance deviation and angle deviation of the preset detection frame and the generated detection frame can be obtained to generate a second loss function based on the distance deviation and the angle deviation; the semantic segmentation model can be used to perform semantic recognition on the feature vector in the preset detection frame to obtain the category of the training object, that is, the above-mentioned preset semantic segmentation result, and a third loss function can be generated by comparing the similarity distance between the category of the training object in the preset semantic segmentation result and the category of the training object in the target semantic segmentation result. The total loss function obtained by the first loss function, the second loss function and the third loss function can be used to adjust the model parameters to make the above-mentioned model more accurate.

[0074] In the above embodiment of the present application, target detection is performed on the training object using the target detection model to obtain the generated detection frame and target semantic segmentation result corresponding to the training object, including: using the feature extraction module to extract features of the training image to obtain a training feature vector; using the semantic segmentation module to perform semantic processing on the training feature vector to obtain the target semantic segmentation result; using the region generation module to process the training feature vector to obtain a training detection frame of a preset angle; using the region conversion module to process the training detection frame of the preset angle to obtain a generated detection frame.

[0075] In an optional embodiment, a training sample can be obtained first, and the feature extraction module can be used to extract features from the training image to obtain a training feature vector. The semantic segmentation module can then be used to perform semantic processing on the training feature vector to obtain a target semantic segmentation result corresponding to the training object, thereby increasing the feature expression ability of the training object. The training feature vector can be processed using a region generation module to obtain a training detection frame of the horizontal angle corresponding to the training object. The training detection frame of the horizontal angle can then be processed using a region conversion module to obtain a generated detection frame consistent with the direction of the target object. The loss function between the detection frames can be determined by comparing the difference between the preset detection frame and the generated detection frame.

[0076] In the above embodiment of the present application, after determining the initial detection frame corresponding to the target object based on the initial feature vector, the method also includes: outputting the initial detection frame; receiving feedback results of the initial detection frame, wherein the feedback results are obtained by modifying the initial detection frame; determining the feedback feature vector corresponding to the feedback result from the fused feature vector; and performing regression processing on the feedback feature vector to obtain the target detection frame.

[0077] In the above embodiment of the present application, outputting the initial detection frame includes: performing object recognition on the image in the initial detection frame to determine the initial category of the target object; obtaining a preset display mode corresponding to the initial category; and outputting the initial detection frame according to the preset display mode.

[0078] The above preset display modes can be set by yourself. The preset display modes are mainly used to distinguish different categories of target objects so that users can quickly distinguish different objects.

[0079] Among them, the target detection frames for different target objects can be displayed in the form of different lines, and the categories of different target objects can be displayed in different fonts; the target detection frames and categories corresponding to different target objects can be displayed in different colors.

[0080] In an optional embodiment, each type of target object may correspond to a different display mode, so as to distinguish target objects of different categories in the target image, making it easier for users to view them.

[0081] For example, for the target detection frame and corresponding category of a building in a building image, a red target detection frame can be displayed, and its category can be displayed in a red font; for the target detection frame of a bridge in a building image, a green target detection frame can be displayed, and its category can be displayed in a green font.

[0082] In another optional embodiment, object recognition can be performed on the image in the initial detection frame to determine the initial category of the target object. After the initial category of the target object is determined, the initial category of the target object can be fed back to the user so that the user can determine whether the initial category of the target object is accurate. If the initial category of the target object is accurate, the initial detection frame can be output according to a preset display method corresponding to the initial category, and the initial detection frames corresponding to target objects of different categories can be distinguished to facilitate user viewing.

[0083] Furthermore, in the case that the initial category of the target object is inaccurate, the user can modify the initial category and obtain a preset display mode corresponding to the modified initial category, and the initial detection frame can be output according to the preset display mode.

[0084] The following combination Figures 3 to 5 A preferred embodiment of the present application is described in detail. The method can be executed by a computer terminal or a server. Figure 3 As shown, the method includes the following steps:

[0085] Step S301, acquiring a remote sensing image;

[0086] Step S302, segmenting and adjusting the remote sensing image to obtain a target image;

[0087] Optionally, the size of the target image may be 1024×1024.

[0088] Step S303, extracting features from the target image using a feature extraction network to obtain an initial feature vector;

[0089] The above feature extraction network can be ResNet50+FPN.

[0090] Step S304: Process the initial feature vector using a region generation module to obtain a detection frame of a preset angle corresponding to the target object;

[0091] The region generation module may be an RPN. The detection frame of the preset angle may be a horizontal detection frame.

[0092] Step S305: Using the region conversion module, the detection frame at the preset angle is rotated to obtain an initial detection frame;

[0093] The region conversion module may be RT. The direction of the initial detection frame may be consistent with the direction of the target object.

[0094] Step S306, performing semantic processing on the initial feature vector to obtain a semantic feature vector;

[0095] Step S307, fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector;

[0096] Step S308, determining the target feature vector corresponding to the initial detection frame from the fused feature vector;

[0097] Step S309: perform regression processing on the target feature vector to obtain a target detection frame.

[0098] like Figure 4 The image shows the target objects in the remote sensing image after being labeled.

[0099] like Figure 5 Shown is a schematic diagram of the image processing process. Based on the rotating object detector, taking ReDet as an example, a semantic segmentation branch is added to the bottom layer of the model, wherein a semantic prediction task can be performed in the semantic segmentation branch to train the semantic segmentation model, and the semantic segmentation model is used to semantically process the initial features in the feature space to obtain a semantic feature vector. Compared with the original ReDet model, ReDet only uses the rotation box as supervision, and the expression ability of the features learned by the model is limited; while the semantic segmentation branch is added in this application by using the rotation box mask (mask annotation) as supervision, rich semantic features from remote sensing images can be obtained. These features are differentiated features that cannot be learned using the rotation box supervision. By adding and fusing the learned semantic features and the rotation RoI features, the feature expression ability of the original model for the rotation RoI can be further enriched and enhanced, thereby further improving the detection accuracy of the rotated target. Multi-task learning is performed through the semantic feature auxiliary branch, that is, the above-mentioned semantic segmentation branch, which can further enhance the feature expression of the rotated object, thereby further improving the detection performance of the rotated target.

[0100] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0102] Example 2

[0103] According to an embodiment of the present application, an image processing method embodiment is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0104] Figure 6 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present invention. Figure 6 As shown, the method may include the following steps:

[0105] Step S602: Acquire a building image.

[0106] The building image includes buildings.

[0107] In an optional embodiment, an image of a building in a city may be captured by a drone or a satellite and transmitted to a server via a network, and the server detects the buildings in the image.

[0108] Step S604: extract features from the building image to obtain an initial feature vector.

[0109] In an optional embodiment, a feature extraction network may be used to extract features from the building image to obtain an initial feature vector.

[0110] Step S606: Perform semantic processing on the initial feature vector to obtain a semantic feature vector.

[0111] In an optional embodiment, the feature vectors of the building's outline, edge, color, texture, shape and other features in the initial feature vector can be semantically processed to obtain a semantic feature vector. Through the semantic feature vector, differentiated features can be learned, thereby improving the feature expression ability of the building. The semantic feature vector and the initial feature vector can be combined to perform building detection on the building image to improve the accuracy of the detection results.

[0112] Step S608 : performing building detection on the building image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the building.

[0113] In an optional embodiment, after obtaining the semantic feature vector, the initial feature vector can be fused with the semantic feature vector to increase the number and richness of building features, thereby improving the expressiveness of the features, and then performing building detection on the building image through the fused feature vector to improve the accuracy of building detection.

[0114] In the above embodiment of the present application, the target detection result includes at least: a target detection frame, which is used to characterize the position of the building in the building image, wherein target detection is performed on the building image based on the initial feature vector and the semantic feature vector, and the target detection result of the building includes: determining the initial detection frame corresponding to the building based on the initial feature vector; fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector; determining the target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain the target detection frame.

[0115] In the above embodiment of the present application, determining the initial detection frame corresponding to the building based on the initial feature vector includes: processing the initial feature vector to obtain a detection frame of a preset angle corresponding to the building; processing the detection frame of the preset angle to obtain the initial detection frame.

[0116] In the above embodiment of the present application, the method also includes: obtaining training samples, wherein the training samples include: training images, and preset detection frames corresponding to the training objects contained in the training images; performing target detection on the training objects using a target detection model to obtain generated detection frames and target semantic segmentation results corresponding to the training objects, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module and a region conversion module, the feature extraction module is used to extract training feature vectors of the training images, the semantic segmentation module is used to generate target semantic segmentation results, the region generation module is used to generate training detection frames of preset angles, and the region conversion module is used to convert training detection frames of preset angles into generated detection frames; generating a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation results; and adjusting the model parameters of the target detection model based on the total loss function.

[0117] In the above embodiment of the present application, target detection is performed on the training object using the target detection model to obtain the generated detection frame and target semantic segmentation result corresponding to the training object, including: using the feature extraction module to extract features of the training image to obtain a training feature vector; using the semantic segmentation module to perform semantic processing on the training feature vector to obtain the target semantic segmentation result; using the region generation module to process the training feature vector to obtain a training detection frame of a preset angle; using the region conversion module to process the training detection frame of the preset angle to obtain a generated detection frame.

[0118] In the above embodiment of the present application, after determining the initial detection frame corresponding to the building based on the initial feature vector, the method also includes: outputting the initial detection frame; receiving feedback results of the initial detection frame, wherein the feedback results are obtained by modifying the initial detection frame; determining the feedback feature vector corresponding to the feedback result from the fused feature vector; and performing regression processing on the feedback feature vector to obtain the target detection frame.

[0119] In the above embodiment of the present application, outputting the initial detection frame includes: performing object recognition on the image in the initial detection frame to determine the initial category of the building; obtaining a preset display mode corresponding to the initial category; and outputting the initial detection frame according to the preset display mode.

[0120] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0121] Example 3

[0122] According to an embodiment of the present application, an image processing method embodiment is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0123] Figure 7 This is a flow chart of an image processing method according to embodiment 3 of the present application. Figure 7 As shown, the method may include the following steps:

[0124] Step S702: The cloud server receives the target image uploaded by the client.

[0125] The target image contains the target object.

[0126] In step S704, the cloud server extracts features from the target image to obtain an initial feature vector.

[0127] Step S706: The cloud server performs semantic processing on the initial feature vector to obtain a semantic feature vector.

[0128] In step S708 , the cloud server performs target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0129] In step S710, the cloud server sends the target detection result to the client.

[0130] In the above embodiment of the present application, the target detection result includes at least: a target detection frame, which is used to characterize the position of the target object in the target image, wherein the cloud server performs target detection on the target image based on the initial feature vector and the semantic feature vector, and obtains the target detection result of the target object, including: the cloud server determines the initial detection frame corresponding to the target object based on the initial feature vector; the cloud server fuses the initial feature vector and the semantic feature vector to obtain a fused feature vector; the cloud server determines the target feature vector corresponding to the initial detection frame from the fused feature vector; the cloud server performs regression processing on the target feature vector to obtain the target detection frame.

[0131] In the above embodiment of the present application, the cloud server determines the initial detection frame corresponding to the target object based on the initial feature vector, including: the cloud server processes the initial feature vector to obtain a detection frame of a preset angle corresponding to the target object; the cloud server processes the detection frame of the preset angle to obtain the initial detection frame.

[0132] In the above embodiment of the present application, the method also includes: the cloud server obtains a training sample, wherein the training sample includes: a training image, and a preset detection frame corresponding to the training object contained in the training image; the cloud server uses the target detection model to perform target detection on the training object, and obtains a generated detection frame and a target semantic segmentation result corresponding to the training object, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module and a region conversion module, the feature extraction module is used to extract the training feature vector of the training image, the semantic segmentation module is used to generate the target semantic segmentation result, the region generation module is used to generate a training detection frame of a preset angle, and the region conversion module is used to convert the training detection frame of the preset angle into a generated detection frame; the cloud server generates a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation result; the cloud server adjusts the model parameters of the target detection model based on the total loss function.

[0133] In the above embodiment of the present application, the cloud server uses the target detection model to perform target detection on the training object, and obtains the generated detection frame and target semantic segmentation results corresponding to the training object, including: the cloud server uses the feature extraction module to extract features from the training image to obtain a training feature vector; the cloud server uses the semantic segmentation module to perform semantic processing on the training feature vector to obtain the target semantic segmentation result; the cloud server uses the region generation module to process the training feature vector to obtain a training detection frame of a preset angle; the cloud server uses the region conversion module to process the training detection frame of the preset angle to obtain a generated detection frame.

[0134] In the above embodiment of the present application, after the cloud server determines the initial detection frame corresponding to the target object based on the initial feature vector, the method also includes: the cloud server outputs the initial detection frame; the cloud server receives the feedback result of the initial detection frame, wherein the feedback result is obtained by modifying the initial detection frame; the cloud server determines the feedback feature vector corresponding to the feedback result from the fused feature vector; the cloud server performs regression processing on the feedback feature vector to obtain the target detection frame.

[0135] In the above embodiment of the present application, the cloud server outputs the initial detection frame, including: the cloud server performs object recognition on the image in the initial detection frame to determine the initial category of the target object; the cloud server obtains the preset display mode corresponding to the initial category; the cloud server outputs the initial detection frame according to the preset display mode.

[0136] In the above embodiment of the present application, the cloud server outputs the target detection result including: the cloud server obtains the preset display mode corresponding to the category of the target object; and the cloud server outputs the target detection result according to the preset display mode.

[0137] Example 4

[0138] According to an embodiment of the present application, an image processing device for implementing the above image processing method is also provided. Figure 8 As shown, the device 800 includes: an acquisition module 802 , an extraction module 804 , a processing module 806 , and a detection module 808 .

[0139] Among them, the acquisition module 802 is used to acquire the target image, wherein the target image contains the target object; the extraction module 804 is used to perform feature extraction on the target image to obtain an initial feature vector; the processing module 806 is used to perform semantic processing on the initial feature vector to obtain a semantic feature vector; the detection module 808 is used to perform target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0140] It should be noted that the acquisition module 802, extraction module 804, processing module 806, and detection module 808 correspond to steps S202 to S208 in Example 1. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0141] In the above embodiments of the present application, the detection module includes: a determination unit, a fusion unit, and a regression unit.

[0142] Among them, the determination unit is used to determine the initial detection frame corresponding to the target object based on the initial feature vector; the fusion unit is used to fuse the initial feature vector and the semantic feature vector to obtain a fused feature vector; the determination unit is also used to determine the target feature vector corresponding to the initial detection frame from the fused feature vector; the regression unit is used to perform regression processing on the target feature vector to obtain the target detection frame.

[0143] In the above embodiment of the present application, the determining unit includes: a processing subunit.

[0144] Among them, the processing subunit is used to process the initial feature vector to obtain a detection frame of a preset angle corresponding to the target object; the processing subunit is also used to process the detection frame of the preset angle to obtain an initial detection frame.

[0145] In the above embodiments of the present application, the device further includes: a generation module and an adjustment module.

[0146] Among them, the acquisition module is also used to obtain training samples, wherein the training samples include: training images, and preset detection frames corresponding to the training objects contained in the training images; the detection module is also used to use the target detection model to perform target detection on the training objects, and obtain the generated detection frames and target semantic segmentation results corresponding to the training objects, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module and a region conversion module, the feature extraction module is used to extract the training feature vector of the training image, the semantic segmentation module is used to generate the target semantic segmentation result, the region generation module is used to generate the training detection frame of the preset angle, and the region conversion module is used to convert the training detection frame of the preset angle into the generated detection frame; the generation module is also used to generate a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation result; the adjustment module is also used to adjust the model parameters of the target detection model based on the total loss function.

[0147] In the above embodiments of the present application, the detection module includes: an extraction unit and a processing unit.

[0148] Among them, the extraction unit is used to use the feature extraction module to extract features from the training image to obtain a training feature vector; the processing unit is used to use the semantic segmentation module to perform semantic processing on the training feature vector to obtain a target semantic segmentation result; the processing unit is also used to use the region generation module to process the training feature vector to obtain a training detection frame of a preset angle; the processing unit is also used to use the region conversion module to process the training detection frame of the preset angle to obtain a generated detection frame.

[0149] In the above embodiments of the present application, the device further includes: an output module, a receiving module, and a determination module.

[0150] Among them, the output module is used to output the initial detection frame; the receiving module is used to receive the feedback result of the initial detection frame, wherein the feedback result is obtained by modifying the initial detection frame; the determination module is used to determine the feedback feature vector corresponding to the feedback result from the fusion feature vector; the processing module is also used to perform regression processing on the feedback feature vector to obtain the target detection frame.

[0151] In the above embodiments of the present application, the output module includes: an identification unit, an acquisition unit, and an output unit.

[0152] Among them, the recognition unit is used to perform object recognition on the image in the initial detection frame to determine the initial category of the target object; the acquisition unit is used to obtain the preset display mode corresponding to the initial category; and the output unit is used to output the initial detection frame according to the preset display mode.

[0153] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0154] Example 5

[0155] According to an embodiment of the present application, an image processing device for implementing the above image processing method is also provided. Figure 9 As shown, the device 900 includes: an acquisition module 902 , an extraction module 904 , a processing module 906 , and a detection module 908 .

[0156] Among them, the acquisition module 902 is used to acquire a building image, wherein the building image contains a building; the extraction module 904 is used to perform feature extraction on the building image to obtain an initial feature vector; the processing module 906 is used to perform semantic processing on the initial feature vector to obtain a semantic feature vector; the detection module 908 is used to perform building detection on the building image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the building.

[0157] It should be noted that the acquisition module 902, extraction module 904, processing module 906, and detection module 908 correspond to steps S602 to S608 in Example 2. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0158] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0159] Example 6

[0160] According to an embodiment of the present application, an image processing device for implementing the above image processing method is also provided. Figure 10 As shown, the device 1000 includes: a receiving module 1002 , an extracting module 1004 , a processing module 1006 , a detecting module 1008 , and a sending module 1010 .

[0161] Among them, the receiving module 1002 is used to receive the target image uploaded by the client through the cloud server, wherein the target image contains the target object; the extraction module 1004 is used to perform feature extraction on the target image through the cloud server to obtain an initial feature vector; the processing module 1006 is used to perform semantic processing on the initial feature vector through the cloud server to obtain a semantic feature vector; the detection module 1008 is used to perform target detection on the target image based on the initial feature vector and the semantic feature vector through the cloud server to obtain a target detection result of the target object; the sending module 1010 is used to send the target detection result to the client through the cloud server.

[0162] It should be noted that the receiving module 1002, extraction module 1004, processing module 1006, detection module 1008, and sending module 1010 described above correspond to steps S702 to S710 in Example 3. The examples and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 3. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0163] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0164] Example 7

[0165] The embodiment of the present application can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0166] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0167] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image processing method: obtaining a target image, wherein the target image contains a target object; performing feature extraction on the target image to obtain an initial feature vector; performing semantic processing on the initial feature vector to obtain a semantic feature vector; and performing target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0168] Optionally, Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of the present application. Figure 11 As shown, the computer terminal 1100 may include: one or more (only one is shown in the figure) processors and a memory.

[0169] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 1100 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0170] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain a target image, wherein the target image contains a target object; perform feature extraction on the target image to obtain an initial feature vector; perform semantic processing on the initial feature vector to obtain a semantic feature vector; and perform target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0171] Optionally, the processor may also execute the program code of the following steps: determining an initial detection frame corresponding to the target object based on the initial feature vector; fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector; determining a target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain a target detection frame.

[0172] Optionally, the processor may further execute program code of the following steps: processing the initial feature vector to obtain a detection frame of a preset angle corresponding to the target object; and processing the detection frame of the preset angle to obtain an initial detection frame.

[0173] Optionally, the processor may also execute the program code of the following steps: obtaining a training sample, wherein the training sample includes: a training image, and a preset detection frame corresponding to the training object contained in the training image; performing target detection on the training object using a target detection model to obtain a generated detection frame and a target semantic segmentation result corresponding to the training object, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module and a region conversion module, the feature extraction module is used to extract a training feature vector of the training image, the semantic segmentation module is used to generate a target semantic segmentation result, the region generation module is used to generate a training detection frame of a preset angle, and the region conversion module is used to convert the training detection frame of the preset angle into a generated detection frame; generating a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation result; and adjusting the model parameters of the target detection model based on the total loss function.

[0174] Optionally, the above-mentioned processor can also execute the program code of the following steps: use the feature extraction module to extract features of the training image to obtain a training feature vector; use the semantic segmentation module to perform semantic processing on the training feature vector to obtain a target semantic segmentation result; use the region generation module to process the training feature vector to obtain a training detection frame of a preset angle; use the region conversion module to process the training detection frame of a preset angle to obtain a generated detection frame.

[0175] Optionally, the processor may also execute the program code of the following steps: outputting an initial detection frame; receiving feedback results of the initial detection frame, wherein the feedback results are obtained by modifying the initial detection frame; determining a feedback feature vector corresponding to the feedback result from the fused feature vector; and performing regression processing on the feedback feature vector to obtain a target detection frame.

[0176] Optionally, the processor may also execute program code of the following steps: performing object recognition on the image in the initial detection frame to determine the initial category of the target object; obtaining a preset display mode corresponding to the initial category; and outputting the initial detection frame according to the preset display mode.

[0177] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain a building image, wherein the building image contains a building; perform feature extraction on the building image to obtain an initial feature vector; perform semantic processing on the initial feature vector to obtain a semantic feature vector; and perform building detection on the building image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the building.

[0178] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: the cloud server receives the target image uploaded by the client, wherein the target image contains the target object; the cloud server extracts features from the target image to obtain an initial feature vector; the cloud server performs semantic processing on the initial feature vector to obtain a semantic feature vector; the cloud server performs target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object; the cloud server sends the target detection result to the client.

[0179] It can be understood by those skilled in the art that Figure 11 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 11It does not limit the structure of the above electronic device. For example, the computer terminal 1100 may also include Figure 11 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 11 Different configurations shown.

[0180] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0181] Example 8

[0182] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method provided in the above embodiment.

[0183] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0184] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a target image, wherein the target image contains a target object; performing feature extraction on the target image to obtain an initial feature vector; performing semantic processing on the initial feature vector to obtain a semantic feature vector; and performing target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object.

[0185] Optionally, the storage medium is also configured to store program code for executing the following steps: determining an initial detection frame corresponding to the target object based on the initial feature vector; fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector; determining a target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain a target detection frame.

[0186] Optionally, the above-mentioned storage medium is also configured to store program codes for executing the following steps: processing the initial feature vector to obtain a detection frame of a preset angle corresponding to the target object; processing the detection frame of the preset angle to obtain an initial detection frame.

[0187] Optionally, the storage medium is further configured to store program code for executing the following steps: obtaining a training sample, wherein the training sample includes: a training image, and a preset detection frame corresponding to the training object contained in the training image; performing target detection on the training object using a target detection model to obtain a generated detection frame and a target semantic segmentation result corresponding to the training object, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module and a region conversion module, the feature extraction module is used to extract a training feature vector of the training image, the semantic segmentation module is used to generate a target semantic segmentation result, the region generation module is used to generate a training detection frame of a preset angle, and the region conversion module is used to convert the training detection frame of the preset angle into a generated detection frame; generating a total loss function based on the preset detection frame, the generated detection frame and the target semantic segmentation result; and adjusting the model parameters of the target detection model based on the total loss function.

[0188] Optionally, the above-mentioned storage medium is also configured to store program codes for executing the following steps: using a feature extraction module to extract features from a training image to obtain a training feature vector; using a semantic segmentation module to perform semantic processing on the training feature vector to obtain a target semantic segmentation result; using a region generation module to process the training feature vector to obtain a training detection frame of a preset angle; using a region conversion module to process the training detection frame of a preset angle to obtain a generated detection frame.

[0189] Optionally, the storage medium is also configured to store program code for executing the following steps: outputting an initial detection frame; receiving feedback results of the initial detection frame, wherein the feedback results are obtained by modifying the initial detection frame; determining a feedback feature vector corresponding to the feedback result from the fused feature vector; and performing regression processing on the feedback feature vector to obtain a target detection frame.

[0190] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: performing object recognition on the image in the initial detection frame to determine the initial category of the target object; obtaining a preset display mode corresponding to the initial category; and outputting the initial detection frame according to the preset display mode.

[0191] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a building image, wherein the building image contains a building; performing feature extraction on the building image to obtain an initial feature vector; performing semantic processing on the initial feature vector to obtain a semantic feature vector; and performing building detection on the building image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the building.

[0192] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: the cloud server receives a target image uploaded by the client, wherein the target image contains a target object; the cloud server performs feature extraction on the target image to obtain an initial feature vector; the cloud server performs semantic processing on the initial feature vector to obtain a semantic feature vector; the cloud server performs target detection on the target image based on the initial feature vector and the semantic feature vector to obtain a target detection result of the target object; the cloud server sends the target detection result to the client.

[0193] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0194] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0196] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0199] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a target image, wherein the target image includes a target object; Performing feature extraction on the target image to obtain an initial feature vector; Performing semantic processing on the initial feature vector to obtain a semantic feature vector; fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector, and performing target detection on the target image based on the fused feature vector to obtain a target detection result of the target object; The target detection result includes at least: a target detection frame, which is used to characterize the position of the target object in the target image. The target detection result of the target object is obtained by performing target detection on the target image based on the fused feature vector, including: determining the initial detection frame corresponding to the target object based on the initial feature vector; determining the target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain the target detection frame.

2. The method according to claim 1, characterized in that Determining an initial detection frame corresponding to the target object based on the initial feature vector includes: Processing the initial feature vector to obtain a detection frame of a preset angle corresponding to the target object; The detection frame at the preset angle is processed to obtain the initial detection frame.

3. The method according to claim 2, characterized in that The method further comprises: Acquire a training sample, wherein the training sample includes: a training image and a preset detection frame corresponding to a training object contained in the training image; Performing target detection on the training object using a target detection model to obtain a generated detection frame and a target semantic segmentation result corresponding to the training object, wherein the target detection model includes: a feature extraction module, a semantic segmentation module, a region generation module, and a region conversion module, wherein the feature extraction module is used to extract a training feature vector of the training image, the semantic segmentation module is used to generate the target semantic segmentation result, the region generation module is used to generate a training detection frame at a preset angle, and the region conversion module is used to convert the training detection frame at the preset angle into the generated detection frame; Generate a total loss function based on the preset detection box, the generated detection box and the target semantic segmentation result; Model parameters of the target detection model are adjusted based on the total loss function.

4. The method according to claim 3, characterized in that Utilizing the target detection model to perform target detection on the training object, obtaining a generated detection frame and target semantic segmentation results corresponding to the training object includes: Using the feature extraction module to extract features from the training image to obtain the training feature vector; Using the semantic segmentation module to perform semantic processing on the training feature vector to obtain the target semantic segmentation result; Processing the training feature vector using the region generation module to obtain a training detection frame of the preset angle; The region conversion module is used to process the training detection frame of the preset angle to obtain the generated detection frame.

5. The method according to claim 1, wherein After determining an initial detection frame corresponding to the target object based on the initial feature vector, the method further includes: Outputting the initial detection frame; receiving a feedback result of the initial detection frame, wherein the feedback result is obtained by modifying the initial detection frame; Determining a feedback feature vector corresponding to the feedback result from the fused feature vector; Regression processing is performed on the feedback feature vector to obtain the target detection frame.

6. The method according to claim 5, characterized in that Outputting the initial detection frame includes: Performing object recognition on the image in the initial detection frame to determine the initial category of the target object; Obtaining a preset display mode corresponding to the initial category; The initial detection frame is output according to the preset display mode.

7. An image processing method, characterized in that: include: Acquire a building image, wherein the building image includes a building; Performing feature extraction on the building image to obtain an initial feature vector; Performing semantic processing on the initial feature vector to obtain a semantic feature vector; fusing the initial feature vector and the semantic feature vector to obtain a fused feature vector, and performing building detection on the building image based on the fused feature vector to obtain a target detection result of the building; The target detection result includes at least: a target detection frame, which is used to characterize the position of the building in the building image. The target detection result of the building is obtained by performing target detection on the building image based on the fused feature vector, including: determining the initial detection frame corresponding to the building based on the initial feature vector; determining the target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain the target detection frame.

8. An image processing method, characterized in that: include: The cloud server receives a target image uploaded by a client, wherein the target image includes a target object; The cloud server performs feature extraction on the target image to obtain an initial feature vector; The cloud server performs semantic processing on the initial feature vector to obtain a semantic feature vector; The cloud server fuses the initial feature vector and the semantic feature vector to obtain a fused feature vector, and performs target detection on the target image based on the fused feature vector to obtain a target detection result of the target object; The cloud server sends the target detection result to the client; The target detection result includes at least: a target detection frame, which is used to characterize the position of the target object in the target image. The target detection result of the target object is obtained by performing target detection on the target image based on the fused feature vector, including: determining the initial detection frame corresponding to the target object based on the initial feature vector; determining the target feature vector corresponding to the initial detection frame from the fused feature vector; and performing regression processing on the target feature vector to obtain the target detection frame.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 8.

10. A computer terminal, characterized in that: include: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the image processing method according to any one of claims 1 to 8 is executed when the program is run.

Images