Object detection method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an object detection method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: receiving a detection request, wherein the detection request comprises an object identifier of a detection object and a detection text; determining a plurality of detection images corresponding to the detection object and the image position of each detection image; analyzing and processing the plurality of detection images and the image position of each detection image to obtain three-dimensional image information of the detection object; performing image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object; according to the detection text and the two-dimensional feature image, a detection result of the detection object is determined, and the detection result is used for indicating the current construction progress of the detection object and the alarm level of the current construction progress. According to the method, the object detection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an object detection method, device, equipment, storage medium and program product. Background Art

[0002] When enterprises are constructing various objects, they need banks to provide financial service support for the operation of various objects. After banks provide financial services for various objects, they need to conduct risk detection on each object.

[0003] In the prior art, the detection object can be the construction of large urban plots, and the construction progress of large urban plots can be detected. The construction of large urban plots usually covers many plots and a wide range. The staff can use drones to take multiple images of the detection object and survey the construction progress of the detection object through multiple images. However, the detection results of the detection object are determined by manually analyzing the captured images, which makes the efficiency of object detection low. Summary of the invention

[0004] The present application provides an object detection method, apparatus, device, storage medium and program product to solve the technical problem of low efficiency of object detection.

[0005] In a first aspect, the present application provides a method for detecting an object, comprising:

[0006] receiving a detection request, wherein the detection request includes an object identifier and a detection text of a detection object;

[0007] Determine a plurality of detection images corresponding to the detection object and an image position of each detection image;

[0008] Analyzing and processing the multiple detection images and the image position of each detection image to obtain three-dimensional image information of the detection object;

[0009] Performing image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object;

[0010] A detection result of the detection object is determined according to the detection text and the two-dimensional feature image, and the detection result is used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

[0011] In a second aspect, the present application provides an object detection device, including a receiving module, a first determination module, an analysis and processing module, an image processing module, and a second determination module:

[0012] The receiving module is used to receive a detection request, wherein the detection request includes an object identifier and a detection text of a detection object;

[0013] The first determination module is used to determine a plurality of detection images corresponding to the detection object and an image position of each detection image;

[0014] The analysis and processing module is used to analyze and process the multiple detection images and the image position of each detection image to obtain the three-dimensional image information of the detection object;

[0015] The image processing module is used to perform image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object;

[0016] The second determination module is used to determine the detection result of the detection object according to the detection text and the two-dimensional feature image, and the detection result is used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0018] The memory stores computer-executable instructions;

[0019] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0022] The object detection method, device, equipment, storage medium and program product provided by the present application can obtain multiple detection images and the image position of each detection image after receiving a detection request, determine the two-dimensional feature image of the detection object, and determine the detection result of the detection object based on the detection text and the two-dimensional feature image. The detection result of the detection object can be determined directly by the detection text and the extracted two-dimensional feature image, without manual analysis and processing, which can improve the efficiency of object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a flow chart of a method for detecting an object provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a flow chart of another object detection method provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a voxel density prediction model provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of a light color preset model provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of an image coding model provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the architecture of a method for detecting an object is provided for an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of an object detection device provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0033] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0037] It should be noted that the object detection methods, devices, equipment, storage media and products provided in the present application can be used in the field of artificial intelligence technology, and can also be used in any field other than artificial intelligence technology. The application fields of the object detection methods, devices, equipment, storage media and products in the present application are not limited.

[0038] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 , including a terminal device 101, a server 102 and a drone 103.

[0039] The terminal device 101 may be a mobile phone, a tablet computer, a desktop computer, a smart wearable device, etc. A detection system may be installed on the terminal device 101. The user may input the object identification and detection text of the detection object in the detection system through the terminal device 101. The terminal device 101 may generate a detection request according to the object identification and detection text of the detection object. The detection request is used to detect the current construction progress of the detection object and determine the alarm level of the detection object. The terminal device 101 may send a detection request to the server 102.

[0040] The server 102 can receive the detection request. The server 102 can send a shooting instruction to the drone 103 according to the detection request. The drone 103 can take videos and photos of the detection object from multiple angles through the camera according to the shooting instruction. The drone 103 can send the taken videos and photos to the server 102.

[0041] The server 102 can determine multiple detection images corresponding to the detection object and the image position of each detection image based on the captured video and photos. The server 102 can analyze and process the multiple detection images and the image position of each detection image to obtain a density feature matrix and a color feature matrix of the detection object, and perform image processing on the density feature matrix and the color feature matrix to obtain a two-dimensional feature image of the detection object.

[0042] The server 102 may determine a detection result of the detection object based on the detection text and the two-dimensional feature image, where the detection result is used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

[0043] In the prior art, the detection object can be the construction of large urban plots, and the construction progress of large urban plots can be detected. The construction of large urban plots usually covers many plots and a wide range. The staff can use drones to take multiple images of the detection object and survey the construction progress of the detection object through multiple images. However, the detection results of the detection object are determined by manually analyzing the captured images, which makes the efficiency of object detection low.

[0044] The object detection method provided by the present application can obtain multiple detection images and the image position of each detection image after receiving a detection request, determine the two-dimensional feature image of the detection object, and determine the detection result of the detection object according to the detection text and the two-dimensional feature image. The detection result of the detection object can be determined directly by the detection text and the extracted two-dimensional feature image, without manual analysis and processing, which can improve the efficiency of object detection.

[0045] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 2 This is a flow chart of an object detection method provided in an embodiment of the present application. Figure 2 , the method may include:

[0047] S201: Receive a detection request.

[0048] The execution subject of the embodiment of the present application may be a server, or may be the detection of an object set in the server. The detection of the object may be implemented by software, or by a combination of software and hardware.

[0049] The detection request may include an object identifier and a detection text of the detection object.

[0050] The detection text is used to indicate the detection content of the detection request. For example, the detection text may be "to what extent a certain building of the detection object has been built so far".

[0051] S202: Determine a plurality of detection images corresponding to the detection object and an image position of each detection image.

[0052] Optionally, multiple initial images and initial videos sent by the drone can be received; the multiple initial images and initial videos are preprocessed to obtain multiple detection images; for any detection image, the image position of the detection image is obtained according to the image pixel coordinates of the detection image and the shooting position of the drone.

[0053] The multiple detection images may include multiple captured images and multiple video frame images.

[0054] Multiple detection images can indicate image information of the detection object in multiple orientations.

[0055] Preprocessing the multiple initial images and initial videos sent by the drone can remove low-quality and repeatedly shot images, which can improve the effectiveness of the detection images and further improve the efficiency of analyzing the detection images.

[0056] S203: Analyze and process the multiple detection images and the image position of each detection image to obtain three-dimensional image information of the detection object.

[0057] The three-dimensional image information may include an image feature vector, a density feature matrix, and a color feature matrix.

[0058] The image feature vector and the density feature matrix can be determined according to the multiple detection images and the image position of each detection image, and the color feature matrix can be determined according to the image feature vector and the shooting angle information of each detection image.

[0059] The density feature matrix may be used to indicate density information of a photographed object in each detection image, and the photographed object may be a detection object.

[0060] The color feature matrix may be used to indicate color information of a photographed object in a detection image, and the photographed object may be a detection object.

[0061] The drone shoots the detection object at different angles, which will produce different shooting lights, resulting in different colors of the detection image after imaging. When extracting the color features of the detection object, the drone's shooting angle information needs to be considered.

[0062] The shooting angle information may include an azimuth and an altitude, and the shooting angle information may be used to indicate the azimuth and altitude of the drone relative to the sun when shooting the detection image.

[0063] The color feature matrix and density feature matrix of the detection object can be extracted from multiple detection images, and the two-dimensional data of the detection image can be modeled as the color feature matrix and density feature matrix in three-dimensional space to determine the two-dimensional feature image of the detection object, thereby improving the accuracy of the two-dimensional feature image.

[0064] S204: Perform image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object.

[0065] The two-dimensional feature image may include pixel values ​​of multiple pixel points.

[0066] For any pixel point, the pixel value corresponding to the pixel point can be obtained through the volume rendering equation. The specific execution process is as follows: the density eigenvalue corresponding to the pixel point is obtained in the density feature matrix, and the color eigenvalue corresponding to the pixel point is obtained in the color feature matrix; according to the density eigenvalue and the color eigenvalue, the pixel point is sampled to obtain multiple sampling values ​​and the sampling weight corresponding to each sampling value; according to the multiple sampling values ​​and each corresponding sampling weight, the pixel value corresponding to the pixel point is obtained.

[0067] The volume rendering equation is used to simulate the process of light propagating in a medium and interacting with the medium. The pixel value of each pixel can be determined through the volume rendering equation based on the density feature matrix and the color feature matrix.

[0068] Each pixel point can be simulated sampled through the density feature matrix and verification feature matrix of the detection object, and then the pixel value corresponding to each pixel point can be determined, which can improve the accuracy of determining the two-dimensional feature image of the detection object.

[0069] The volume rendering equation can be shown as follows:

[0070]

[0071]

[0072] Among them, any light ray r at any position p passing through the scene forms the expected color C of the image, in the density field A ray r is emitted from the ray, and any point on the ray is x, then the ray can be expressed as r(p)=x+pd, where d is a 3D Cartesian unit vector, the ray is bounded by points pn to pf, and T(p) represents the probability that the ray propagates from pn to p without disappearing.

[0073] The multiple sampling values ​​may include multiple first sampling values ​​and multiple second sampling values; according to the density feature value and the color feature value, the first layered sampling is performed on the pixel points to obtain multiple first sampling values ​​and sampling weights of the multiple first sampling values; according to the sampling weights of the multiple first sampling values, the second layered sampling is performed on the pixel points to obtain multiple second sampling values ​​and sampling weights of each second sampling value. The second layered sampling can accurately sample the pixel points.

[0074] The first stratified sampling can be stratified coarse sampling. For any pixel point, the light between the drone shooting position and the pixel point of the detection object can be evenly divided into N areas, and random uniform sampling is performed in each area to determine N sampling points. Based on the above volume rendering equation, the estimated pixel value of the pixel point can be determined according to the N sampling points. The formula for the discrete quadrature of the estimated pixel value is as follows:

[0075]

[0076] in, Used to indicate the estimated pixel value of the sampling point. It is used to indicate the non-disappearance probability of the i-th sampling point. Used to indicate the density of the i-th sampling point, Used to indicate the sampling value of the i+1th sampling point. Used to indicate the sampling value of the i-th sampling point. Used to indicate the pixel value of the i-th sampling point.

[0077] In addition, the voxel density between each sampling point can be approximately estimated and color c are both constants, so the weighted form of the estimated pixel value of the rough network after the first coarse sampling rendering is calculated as follows:

[0078]

[0079] By transforming the above two formulas, the sampling weight of each sampling point can be determined by the following formula:

[0080]

[0081] The first stratified sampling can be stratified coarse sampling. During the coarse sampling process, all areas are randomly and uniformly sampled. Usually, free space and occluded areas that are not related to the area of ​​interest are repeatedly sampled. The second stratified sampling can be used to perform secondary fine sampling of points on each ray to increase the sampling of areas with high importance, thereby achieving importance sampling.

[0082] The second stratified sampling method can be a rejection sampling method. The rejection sampling method can refer to the following steps:

[0083] (1) The sampling weights of the multiple sampling points obtained in the first layer sampling are normalized to obtain the probability density function F(x). The probability density function can be used to indicate the weight function between the drone shooting position and the pixel point. The closer to the pixel point, the higher the weight. x is used to indicate any sampling point between the drone shooting position and the pixel point.

[0084] (2) Set a constant b so that for all x, f(x) / F(x)<=b is satisfied. The constant b is used to divide the weights of the sampling points. The x that satisfies f(x) / F(x)<=b is used to indicate that the pixel is close and can be sampled, so that the closer the pixel is, the more sampling points there are.

[0085] (3) Generate a random value t from the distribution F(x) and generate a random value u from the (0,1) uniform distribution function.

[0086] (4) If satisfied , then accept t and substitute x=t, otherwise reject t.

[0087] (5) Repeat steps (3)-(4) until multiple second sampling values ​​are sampled.

[0088] A plurality of first sampling values, a sampling weight corresponding to each first sampling value, a plurality of second sampling values, and a sampling weight of each second sampling value may be determined, and the pixel value of the pixel point may be obtained according to the following formula:

[0089]

[0090] in, Used to indicate the pixel value of a pixel. Used to indicate the number of the first sampling points, Used to indicate the number of second adoption points, Used to indicate the sampling weight of the jth sampling point, Used to indicate the sampling value of the jth sampling point.

[0091] After determining the pixel value of each pixel, a two-dimensional feature image of the detected object can be obtained. The efficiency of the rendering model and the quality of the generated image can be improved through the first layer sampling and the second layer sampling.

[0092] S205: Determine a detection result of the detection object according to the detection text and the two-dimensional feature image.

[0093] The detection text and the two-dimensional feature image can be input into the question-answering model to obtain the detection result of the detection object. The question-answering model can include a multimodal encoder, a modal connection, and a large language model.

[0094] Optionally, visual extraction processing is performed on the two-dimensional feature image to obtain visual features; text feature processing is performed on the determined detection text to obtain text features; and a detection result corresponding to the detection request is determined based on the visual features and the text features.

[0095] The detection result of the detection object can be determined by extracting visual features and text features, which can improve the accuracy of the detection result.

[0096] The detection result can be used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

[0097] The detection result can be sent to the terminal device. If the alarm level of the detection result is greater than the preset alarm level, a prompt message can be sent to the terminal.

[0098] The object detection method provided in this embodiment can obtain multiple detection images and the image position of each detection image after receiving a detection request, determine the two-dimensional feature image of the detection object, and determine the detection result of the detection object according to the detection text and the two-dimensional feature image. The detection result of the detection object can be determined directly by the detection text and the extracted two-dimensional feature image, without manual analysis and processing, which can improve the efficiency of object detection.

[0099] Figure 3 This is a flow chart of another object detection method provided in an embodiment of the present application. Figure 3 , the method may include:

[0100] S301: Receive a detection request.

[0101] The execution process of S301 can refer to the execution process of S201, which will not be repeated here.

[0102] S302: Receive multiple initial images and initial videos sent by the drone.

[0103] The drone can receive a shooting instruction sent by the server, and the shooting instruction can include the geographic location of the detection object. The drone can shoot the detection object according to the shooting instruction to obtain multiple initial images and initial videos.

[0104] In order to ensure that the photos taken by the drone can completely cover the entire detection object, the initial image can be an image of the detection object in multiple orientations, and multiple initial images are taken in each orientation. In addition, the initial video can also be one or more.

[0105] S303: Preprocess the multiple initial images and the initial video to obtain multiple detection images.

[0106] The multiple detection images may include multiple captured images and multiple video frame images.

[0107] Furthermore, the multiple initial images can be screened and processed by the peak signal-to-noise ratio of each initial image and the quality index of each image to obtain multiple captured images; the initial video can be sampled and processed by a preset sampling frequency to obtain multiple video frame images.

[0108] Peak Signal-to-Noise Ratio (PSNR) can be an indicator for measuring image or video quality, with the unit being decibel (dB).

[0109] In some possible embodiments, for any initial image, the initial image may be an RGB image, where R is the abbreviation of red, G is the abbreviation of green, and B is the abbreviation of blue. The initial image may include an R channel, a G channel, and a B channel. Noise processing may be performed on the initial pixel value of each channel to obtain the intermediate pixel value of each channel; the mean square error of the intermediate pixel value of each channel is determined, and the average of the mean square errors of the intermediate pixel values ​​of each channel is determined as the PSNR value.

[0110] The quality index may be a Universal Image Quality Index (UIQI), which is an indicator for evaluating image quality.

[0111] An initial image whose peak signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio can be determined as an intermediate image, so as to determine multiple intermediate images from multiple initial images; the similarity of the quality index between each intermediate image can be determined, and if there are any two intermediate images whose similarity is greater than a preset similarity, one of the intermediate images from the multiple intermediate images is deleted to obtain multiple captured images.

[0112] In some possible embodiments, the preset signal-to-noise ratio may be 30 dB, and the preset similarity may be 0.7.

[0113] If there are multiple initial videos, the VMAF value of each initial video can be determined through Video Multi-Method Assessment Fusion (VMAF), and the initial videos with VMAF values ​​less than a preset VMAF value are deleted. The preset VMAF value may be 80.

[0114] For any initial video, the initial video is sampled at a preset sampling frequency to obtain a plurality of video frame images.

[0115] In some possible embodiments, the preset sampling frequency may be 1 second, and the initial video may be sampled at a sampling rate of every 1 second to obtain a plurality of video frame images.

[0116] In some possible embodiments, the image sizes of the multiple video frame images may be unified, and the resolutions of the multiple video frame images may be improved by bilinear interpolation.

[0117] By screening multiple initial images and sampling the initial video to obtain multiple detection images, the effectiveness of the multiple detection images can be improved.

[0118] S304: For any detection image, the image position of the detection image is obtained according to the image pixel coordinates of the detection image and the shooting position of the drone.

[0119] Determine the image position of the detection image ( , ), you can refer to the following formula:

[0120]

[0121]

[0122] Among them, the image pixel coordinates of the detection image are ( , ), the shooting position of the drone is ( , ), To detect the resolution of the image in the x direction, It is the resolution of the detected image in the y direction. pixel is pixel, longitude is longitude, latitude is latitude, resolution is resolution, and geo is geography.

[0123] S305: Determine a density feature matrix according to the multiple detection images and the image position of each detection image.

[0124] The multiple detection images and the image position of each detection image can be processed through the voxel density prediction model to obtain a density feature matrix.

[0125] The voxel density prediction model may include a first feature model and a first perceptron model.

[0126] The first feature model can be used to perform data processing on multiple detection images to obtain a first feature vector; the image position of each detection image can be normalized to obtain a position code; the first feature vector and the position code can be spliced ​​to obtain a first splicing vector; the first splicing vector can be feature extracted through the first perceptron model to obtain an image feature vector; and the image feature vector can be data processed through a normalized exponential function (softmax layer) to obtain a density feature vector.

[0127] Figure 4 This is a schematic diagram of the structure of a voxel density prediction model provided in an embodiment of the present application. Figure 4 , multiple detection images and the image position of each detection image can be input into the voxel density prediction model. The first feature model can perform feature extraction processing on each detection image to obtain a first feature vector. The voxel density prediction model can normalize the image position to obtain a position code, and splice the first feature vector and the position code of each detection image to obtain a first splicing vector. The first splicing vector can be input into the first perceptron model to obtain an image feature vector, and the image feature vector is processed through the softmax layer of the voxel density prediction model to obtain a density feature matrix corresponding to the detection image.

[0128] The first feature model may be a feature pyramid model, wherein a residual network (ResidualNetwork, ResNet-101) may be used as a backbone network of the feature pyramid model, and 101 is used to indicate that the residual network includes 101 layers.

[0129] The first perceptron model may be a plurality of perceptrons, each of which includes a plurality of processing layers.

[0130] The image feature vector may be a 256-dimensional feature vector.

[0131] The first splicing vector is obtained by splicing the position code of each detection image in the first feature vector, and the density feature vector is obtained by extracting the features of the first splicing vector through the first perceptron model. The influence of the shooting position of the detection object is taken into account, which can improve the accuracy of the density feature vector.

[0132] In some possible embodiments, the first feature model performs feature extraction processing on multiple detection images to obtain a first feature vector, which can be referred to as the following execution process:

[0133] Step 1: Bottom-up convolution.

[0134] The detection image may be convolved from bottom to top in the following manner: the detection image may be downsampled N times to obtain N feature maps, where N is an integer greater than or equal to 1.

[0135] Specifically, the step size and sampling times corresponding to the i-th downsampling process can be obtained; according to the step size and the sampling times, the detection image is downsampled for the i-th time to obtain the i-th feature map, where i is 1, 2, ..., N.

[0136] For example, N is 4, and the N feature maps are the first feature map D1, the second feature map D2, the third feature map D3, and the fourth feature map D4. With a step size of 4, the detection image is downsampled twice to obtain the first feature map, and the size of the first feature map is (H / 4, W / 4, 64); with a step size of 8, the detection image is downsampled 3 times to obtain the second feature map, and the size of the second feature map is (H / 8, W / 8, 128); with a step size of 16, the detection image is downsampled 4 times to obtain the third feature map, and the size of the third feature map is (H / 16, W / 16, 256); with a step size of 32, the detection image is downsampled 5 times to obtain the fourth feature map, and the size of the fourth feature map is (H / 32, W / 32, 512). The underlying texture and color information of the detection image is retained in the first and second feature maps, and the semantic information of the detection image is retained in the third and fourth feature maps.

[0137] Step 2: Top-down convolution.

[0138] The N feature maps can be convolved from top to bottom in the following manner: the Nth feature map can be convolved 1*1 to obtain the Nth upsampled feature map; the Nth upsampled feature map can be upsampled by nearest neighbor interpolation to obtain the first intermediate feature map; the N-1th feature map can be convolved 1*1 to obtain the second intermediate feature map; the first intermediate feature map and the second intermediate feature map can be matrix-added to obtain the N-1th upsampled feature map; until N is 2, the first upsampled feature map can be obtained.

[0139] For example, N is 4, and a 1*1 convolution is performed on the fourth feature map D4 to obtain a fourth upsampled feature map U4 of size (H / 32, W / 32, 256). Then, the fourth upsampled feature map U4 is upsampled by applying the nearest neighbor interpolation to obtain a first intermediate feature map, and the size of the first intermediate feature map is (H / 16, W / 16, 256). A 1*1 convolution is performed on the third feature map D3 to obtain a second intermediate feature map of size (H / 16, W / 16, 256). The first intermediate feature map and the second intermediate feature map are matrix-added to obtain a third upsampled feature map U3 of size (H / 16, W / 16, 256). Similarly, two more upsamplings are performed to obtain a second upsampled feature map U2 and a first upsampled feature map U1. The size of the second upsampled feature map U2 is (H / 8, W / 8, 256), and the size of the third upsampled feature map U1 is (H / 4, W / 4, 256).

[0140] Step 3: Feature splicing output.

[0141] Perform 3*3 convolution processing on the N upsampled feature maps to obtain N convolution feature maps; obtain N weight coefficients corresponding to the N convolution feature maps; multiply the i-th weight coefficient by the i-th convolution feature map to obtain the i-th weight feature map; splice the N weight feature maps in the channel direction to obtain the first feature vector.

[0142] For example, a 3*3 convolution is performed on U1, U2, U3, and U4 respectively to obtain four convolution feature maps, namely, convolution feature Figure 1-4 . Get the 4 weight coefficients corresponding to each convolution feature map, which are weight coefficients 1-4. Multiply each first convolution feature map by its corresponding weight coefficient to obtain 4 weight feature maps, which are weight feature Figure 1-4 . The four weight feature maps are concatenated in the channel direction to obtain the first feature vector Y.

[0143] Performing 3*3 convolution on the upsampled feature map can reduce the aliasing effect during the upsampling process.

[0144] Since a deep learning network may have a tendency to favor low-frequency features, the image position of the detection image can be encoded, and the position information of the detection image can be input into the network to improve the performance of the deep network in terms of high-frequency change features. The following is a specific description of the execution process of determining the position encoding of multiple detection images.

[0145] The Fourier series equation can be used as a high-frequency encoding function. After normalizing multiple image positions, the encoding process is performed using the Fourier series equation. The following formula can be used:

[0146]

[0147] in, Used to indicate the position code of the detected image, Used to indicate the image location, Used to indicate the normalized position of the first image position. There are M detection images in total, and M can be an integer greater than or equal to 2.

[0148] S306: Determine a color feature matrix according to the image feature vector and the shooting angle information of each detection image.

[0149] The color feature matrix can be obtained by processing multiple density feature matrices and the shooting angle information of each detection image through a light color preset model.

[0150] The preset light color model may include a second perceptron model.

[0151] Specifically, the shooting angle information of each detection image can be normalized to obtain the angle position code of each detection image; the image feature vector and the angle position code of each detection image can be spliced ​​to obtain a second spliced ​​vector; and the second spliced ​​vector can be feature extracted through a second perceptron model to obtain a color feature matrix.

[0152] Figure 5 This is a schematic diagram of the structure of a light color preset model provided in an embodiment of the present application. Figure 5 , the image feature vector and the shooting angle information of each detection image can be input into the light color preset model. The shooting angle information of each detection image can be normalized and position encoded to obtain the angle position encoding of each detection image. The density feature matrix and the angle position encoding of each detection image are spliced ​​to obtain a second spliced ​​vector. The second spliced ​​vector is input into the second perceptron model to obtain a second feature vector, and the second feature vector can be processed by the fully connected layer of the light color preset model to obtain a color feature matrix.

[0153] The second perceptron model may be a plurality of perceptrons, each of which includes a plurality of processing layers.

[0154] The Fourier series equation can be used as a high-frequency encoding function. After normalizing the information of multiple shooting angles, the encoding process is performed using the Fourier series equation. The following formula can be used:

[0155]

[0156] in, Used to indicate the angle position code of the detected image, Used to indicate shooting angle information. It is used to indicate the normalized position of the first shooting angle information. There are M detection images in total, and M can be an integer greater than or equal to 2.

[0157] The angle position code of each detection image is spliced ​​in the image feature vector to obtain a second spliced ​​vector. The second spliced ​​vector is subjected to feature extraction processing through a second perceptron model to obtain a color feature matrix, thereby improving the accuracy of the color feature matrix.

[0158] S307 , performing image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object.

[0159] The execution process of S301 can refer to the execution process of S204, which will not be repeated here.

[0160] S308: Perform visual extraction processing on the two-dimensional feature image to obtain visual features.

[0161] The two-dimensional feature image can be visually extracted and processed through the image coding model to obtain visual features.

[0162] Figure 6 This is a schematic diagram of the structure of the image coding model provided in the embodiment of the present application. Figure 6 After the two-dimensional feature image is input into the image coding model, the image coding model can perform size conversion processing on the two-dimensional feature image to obtain a preset size image, and the image coding model can perform image block processing on the preset size image to obtain multiple block images. The image coding model can perform linear transformation processing on multiple block images to obtain a label sequence vector. The label sequence vector is processed by multiple encoder blocks, wherein the encoder block may include a normalization layer, a multi-head attention mechanism layer, and a batch normalization layer (Dropath layer). After each encoder block processes the input feature, an output feature is obtained, and the input feature and the output feature are superimposed and input into the next encoder block. The output feature of the last encoder block is processed by the normalization layer and the perceptron layer to obtain a visual feature.

[0163] S309: Perform text feature processing on the determined detected text to obtain text features.

[0164] The text feature processing of the detected text can be performed through the text editor model to obtain the text feature. The structure of the text editor model can be found in the image encoding model and will not be described in detail here.

[0165] S310: Determine a detection result corresponding to the detection request according to the visual features and the text features.

[0166] Optionally, determine the language features corresponding to the visual features; perform text processing on the language features and text features through a language model to obtain a detection result.

[0167] The visual features can be processed through the modal connection model to obtain language features, wherein the modal connection model can include two perceptron layers. The modal connection model can project the visual features into the language space, align the visual features with the text features, and improve the effectiveness of language model processing.

[0168] The language model can be a pre-trained large model, through which text features and language features can be converted into probabilities in the vocabulary, and index values ​​of words with high probability values ​​can be output, ultimately forming detailed text review detection results.

[0169] The object detection method provided in the embodiment of the present application obtains multiple detection images of the detection object after receiving the detection request, determines the three-dimensional image information based on the multiple detection images, and can display more features of the detection object through the three-dimensional image information. Through the three-dimensional image information, the two-dimensional feature image corresponding to the detection is determined, which can improve the accuracy of the two-dimensional feature image. At the same time, the text detection result of the detection object can be directly displayed to the user without the user analyzing the image, which can improve the efficiency of detection.

[0170] Figure 7 The present invention provides a schematic diagram of an object detection method. Figure 7 After receiving the detection request, multiple initial images and initial videos of the detection object corresponding to the detection request can be obtained, and the multiple initial images and initial videos can be preprocessed to obtain multiple detection images and the image position corresponding to each detection image.

[0171] The image feature vector and the density feature matrix can be determined according to the multiple detection images and the image position of each detection image, and the color feature matrix can be determined according to the image feature vector and the shooting angle information of each detection image. The density feature matrix and the color feature matrix are processed to obtain a two-dimensional feature image.

[0172] The detection result of the detection object can be determined based on the detection text and the two-dimensional feature image corresponding to the detection request. The detection result can be used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

[0173] Figure 8 This is a schematic diagram of the structure of an object detection device provided in an embodiment of the present application. Figure 8 The object detection device may include a receiving module 11, a first determining module 12, an analyzing and processing module 13, an image processing module 14 and a second determining module 15:

[0174] The receiving module 11 is used to receive a detection request, where the detection request includes an object identifier and a detection text of a detection object;

[0175] The first determination module 12 is used to determine a plurality of detection images corresponding to the detection object and an image position of each detection image;

[0176] The analysis and processing module 13 is used to analyze and process the multiple detection images and the image position of each detection image to obtain three-dimensional image information of the detection object;

[0177] The image processing module 14 is used to perform image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object;

[0178] The second determination module 15 is used to determine the detection result of the detection object according to the detection text and the two-dimensional feature image, and the detection result is used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

[0179] The object detection device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0180] In some possible embodiments, the three-dimensional image information includes an image feature vector, a density feature matrix, and a color feature matrix; the analysis and processing module 13 is specifically used for:

[0181] Determine an image feature vector and a density feature matrix according to the plurality of detection images and the image position of each detection image;

[0182] The color feature matrix is ​​determined according to the image feature vector and the shooting angle information of each detection image.

[0183] In some possible embodiments, the analysis and processing module 13 is specifically used for:

[0184] Performing data processing on a plurality of detection images through a first feature model to obtain a first feature vector;

[0185] Normalize the image position of each detection image to obtain a position code;

[0186] Concatenate the first feature vector and the position code to obtain a first concatenation vector;

[0187] The first concatenated vector is subjected to feature extraction processing through the first perceptron model to obtain an image feature vector, and the image feature vector is subjected to data processing through a normalized exponential function to obtain a density feature matrix.

[0188] In some possible embodiments, the analysis and processing module 13 is specifically used for:

[0189] Normalize the shooting angle information of each detection image to obtain the angle position code of each detection image;

[0190] The image feature vector and the angle position code of each detection image are spliced ​​to obtain a second splicing vector;

[0191] The second concatenated vector is subjected to feature extraction processing through the second perceptron model to obtain a color feature matrix.

[0192] In some possible embodiments, the two-dimensional feature image includes pixel values ​​of multiple pixel points; the image processing module 14 is specifically used for:

[0193] Obtaining density eigenvalues ​​corresponding to the pixel points in the density eigenvalue matrix, and obtaining color eigenvalues ​​corresponding to the pixel points in the color eigenvalue matrix;

[0194] According to the density eigenvalue and the color eigenvalue, the pixel points are sampled to obtain multiple sampling values ​​and a sampling weight corresponding to each sampling value;

[0195] According to the multiple sampling values ​​and each corresponding sampling weight, a pixel value corresponding to the pixel point is obtained.

[0196] In some possible embodiments, the image processing module 14 is specifically used for:

[0197] The plurality of sampling values ​​include a plurality of first sampling values ​​and a plurality of second sampling values;

[0198] Performing first hierarchical sampling on the pixel points according to the density characteristic value and the color characteristic value to obtain a plurality of first sampling values ​​and sampling weights of the plurality of first sampling values;

[0199] According to the sampling weights of the multiple first sampling values, the pixel points are subjected to second layered sampling to obtain multiple second sampling values ​​and the sampling weight of each second sampling value. The second layered sampling performs precise sampling on the pixel points.

[0200] In some possible embodiments, the second determining module 15 is specifically configured to:

[0201] Perform visual extraction processing on the two-dimensional feature image to obtain visual features;

[0202] Performing text feature processing on the determined detected text to obtain text features;

[0203] According to the visual features and text features, the detection result corresponding to the detection request is determined.

[0204] In some possible embodiments, the second determining module 15 is specifically configured to:

[0205] Determine the language features that correspond to the visual features;

[0206] Through the language model, the language features and text features are processed to obtain the detection results.

[0207] In some possible embodiments, the first determining module 11 is specifically configured to:

[0208] receiving a plurality of initial images and initial videos sent by the drone;

[0209] Preprocessing the multiple initial images and the initial video to obtain multiple detection images, where the multiple detection images include multiple captured images and multiple video frame images;

[0210] For any detection image, the image position of the detection image is obtained according to the image pixel coordinates of the detection image and the shooting position of the drone.

[0211] In some possible embodiments, the first determining module 11 is specifically configured to:

[0212] The multiple initial images are screened and processed by the peak signal-to-noise ratio of each initial image and the similarity between each image to obtain multiple captured images;

[0213] The initial video is sampled by a preset sampling frequency to obtain a plurality of video frame images.

[0214] The object detection device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0215] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 The electronic device 20 may include a processor 21 and a memory 22. Exemplarily, the processor 21 and the memory 22 are interconnected via a bus 23.

[0216] The memory 22 stores computer executable instructions;

[0217] The processor 21 executes the computer-executable instructions stored in the memory 22, so that the processor 21 executes the object detection method shown in the above method embodiment.

[0218] Accordingly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the object detection method of the above method embodiment.

[0219] Accordingly, an embodiment of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the object detection method shown in the above method embodiment.

[0220] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0222] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0223] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0224] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0225] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0226] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0227] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0228] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for detecting an object, characterized in that: include: receiving a detection request, wherein the detection request includes an object identifier and a detection text of a detection object; Determine a plurality of detection images corresponding to the detection object and an image position of each detection image; Analyzing and processing the multiple detection images and the image position of each detection image to obtain three-dimensional image information of the detection object; Performing image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object; A detection result of the detection object is determined according to the detection text and the two-dimensional feature image, and the detection result is used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

2. The method according to claim 1, characterized in that The three-dimensional image information includes an image feature vector, a density feature matrix and a color feature matrix; Analyzing and processing the multiple detection images and the image position of each detection image to obtain three-dimensional image information of the detection object includes: Determine the image feature vector and the density feature matrix according to the multiple detection images and the image position of each detection image; The color feature matrix is ​​determined according to the image feature vector and the shooting angle information of each detection image.

3. The method according to claim 2, characterized in that Determining the image feature vector and the density feature matrix according to the multiple detection images and the image position of each detection image includes: Performing data processing on the plurality of detection images through a first feature model to obtain a first feature vector; Normalizing the image position of each detection image to obtain a position code; Concatenating the first feature vector and the position code to obtain a first concatenated vector; The first splicing vector is subjected to feature extraction processing through a first perceptron model to obtain the image feature vector, and the image feature vector is subjected to data processing through a normalized exponential function to obtain the density feature matrix.

4. The method according to claim 2, characterized in that: Determining the color feature matrix according to the image feature vector and the shooting angle information of each detection image includes: Normalizing the shooting angle information of each detection image to obtain the angle position code of each detection image; Splicing the image feature vector and the angle position code of each detection image to obtain a second splicing vector; The second concatenated vector is subjected to feature extraction processing by using a second perceptron model to obtain the color feature matrix.

5. The method according to any one of claims 1 to 4, characterized in that: The two-dimensional feature image includes pixel values ​​of a plurality of pixel points; For any pixel point; performing image processing on the density feature matrix and the color feature matrix to obtain a pixel value corresponding to the pixel point, including: Obtaining density feature values ​​corresponding to the pixel points in the density feature matrix, and obtaining color feature values ​​corresponding to the pixel points in the color feature matrix; According to the density feature value and the color feature value, the pixel point is sampled to obtain a plurality of sampling values ​​and a sampling weight corresponding to each sampling value; A pixel value corresponding to the pixel point is obtained according to the multiple sampling values ​​and each corresponding sampling weight.

6. The method according to claim 5, characterized in that The pixel point is sampled according to the density feature value and the color feature value to obtain a plurality of sampling values ​​and a sampling weight corresponding to each sampling value, including: The plurality of sampling values ​​include a plurality of first sampling values ​​and a plurality of second sampling values; Performing first hierarchical sampling on the pixel points according to the density feature value and the color feature value to obtain a plurality of first sampling values ​​and sampling weights of the plurality of first sampling values; According to the sampling weights of the multiple first sampling values, the pixel points are subjected to second layered sampling to obtain multiple second sampling values ​​and a sampling weight for each second sampling value. The second layered sampling accurately samples the pixel points.

7. The method according to any one of claims 1 to 6, characterized in that: Determining a detection result of the detection object according to the detection text and the two-dimensional feature image includes: Performing visual extraction processing on the two-dimensional feature image to obtain visual features; Performing text feature processing on the determined detected text to obtain text features; A detection result corresponding to the detection request is determined according to the visual feature and the text feature.

8. The method according to claim 7, characterized in that Determining, according to the visual feature and the text feature, a detection result corresponding to the detection request, including: Determining a language feature corresponding to the visual feature; The language features and the text features are processed by a language model to obtain the detection result.

9. The method according to any one of claims 1 to 8, characterized in that: Determining a plurality of detection images corresponding to the detection object and an image position of each detection image includes: receiving a plurality of initial images and initial videos sent by the drone; Preprocessing the multiple initial images and the initial video to obtain the multiple detection images, wherein the multiple detection images include multiple captured images and multiple video frame images; For any detection image, the image position of the detection image is obtained according to the image pixel coordinates of the detection image and the shooting position of the drone.

10. The method according to claim 9, characterized in that Preprocessing the multiple initial images and the initial video to obtain the multiple detection images includes: The plurality of initial images are screened by using the peak signal-to-noise ratio of each initial image and the similarity between each image to obtain the plurality of captured images; The initial video is sampled by a preset sampling frequency to obtain the multiple video frame images.

11. A device for detecting an object, characterized in that: It includes a receiving module, a first determining module, an analyzing and processing module, an image processing module and a second determining module: The receiving module is used to receive a detection request, wherein the detection request includes an object identifier and a detection text of a detection object; The first determination module is used to determine a plurality of detection images corresponding to the detection object and an image position of each detection image; The analysis and processing module is used to analyze and process the multiple detection images and the image position of each detection image to obtain the three-dimensional image information of the detection object; The image processing module is used to perform image processing on the three-dimensional image information to obtain a two-dimensional feature image of the detection object; The second determination module is used to determine the detection result of the detection object according to the detection text and the two-dimensional feature image, and the detection result is used to indicate the current construction progress of the detection object and the alarm level of the current construction progress.

12. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 10 when being executed by a processor.

Images