A data mining system, method and device based on joint text and image information

Abstract

The application provides a kind of data mining system, method and device of text and image information combination.First, the data pool feature of target object in data pool is extracted, and the data pool feature is extracted based on the single view image of part or all of the target object.Second, the query feature of the object to be queried in the query information is extracted, and the query feature is obtained by fusing the text semantic feature and the query image feature.Finally, the feature similarity of the data pool feature and the query feature is determined, and the mining result related to the object to be queried in the target object is output according to the threshold setting of the feature similarity.Through fusing the data of two modalities of image and text as query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image are reduced.Through fusing the image data of multi-angle camera equipment as data pool data, the situation that a single camera equipment cannot cover the complete target object is avoided, and the integrity of the target object feature in the data pool is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data mining, and particularly relates to a data mining system, method and device based on joint of image and text information. BACKGROUND

[0002] In the automatic driving business, a large amount of road environment measurement data can be obtained by a data collection vehicle and stored in a cloud database. Usually, part of specific sample data (such as special-shaped vehicles, small obstacles, etc.) is selected from the database to train and iterate the artificial intelligence model. Since the overall sample size of the database is large, in order to quickly and accurately mine the required data samples, a high-performance data mining tool needs to be developed.

[0003] In recent years, the image-text matching task has gradually become popular in the fields of artificial intelligence and machine learning. The multi-modal image-text matching task system needs to focus on both text and image as two different data modalities as research objects and accurately extract the features of the text and image. In the process of data mining through image-text matching, the data faced by the retrieval is massive, and there is often a correlation between the multi-modal data. Therefore, how to accurately perform image-text mutual retrieval is a problem to be solved at present. SUMMARY

[0004] To solve the above problems, the present application provides a data mining method and system based on joint of image and text information. The data mining method shown in the present application can reduce the semantic uncertainty contained in the text and the texture uncertainty contained in the image by fusing the image data and the text data as query information, so that the query target feature of the query information is more accurate. At the same time, by fusing the image data captured by multiple perspective camera devices as data pool data, the situation that a single camera device cannot cover the complete target object can be avoided, and the integrity of the data pool target feature is improved.

[0005] In a first aspect, a data mining method is provided. First, data pool features of a target object in a data pool are extracted, wherein the data pool features are extracted based on part or all of the single-view images of the target object. Second, query features of a query object in query information are extracted, wherein the query features are obtained by fusing text semantic features and query image features, the text semantic features are extracted from a query text, the query text records the text for describing the query object, and the query image features are extracted from a query image, the query image includes part or all of the query object. Finally, a feature similarity between the data pool features and the query features is determined, and a mining result is output according to a threshold setting of the feature similarity. Through the data mining method provided by the present application, not only the query target feature of the query information can be more accurate, but also the integrity of the data pool target feature can be improved.

[0006] Regarding the data pool feature mentioned above, it can be obtained by fusing multiple single-view image features, which are obtained by feature extraction on multiple single-view images respectively. The multiple single-view images include images captured at different angles. By extracting multiple single-view images, more fusion materials can be ensured for the image surround view feature.

[0007] Next, a method for extracting data pool features is introduced. First, multiple single-view image features are obtained by feature extraction on multiple single-view images respectively. Second, the overlapping areas between multiple single-view images are calculated, as well as the area proportions of the overlapping areas in single-view images. Third, based on the overlapping areas between multiple single-view images and the area proportions of the overlapping areas in single-view images, multiple single-view image features are fused to obtain image surround view features. Finally, according to the position information of the target object, which has been labeled in multiple single-view images, the image surround view features within the target object region are taken as data pool features. By fusing image data captured by multiple angle cameras as data pool data, the situation that a single camera cannot cover the complete target object can be avoided.

[0008] Next, a method for extracting single-view image features is introduced. First, each single-view image is divided into multiple first grid images, and the feature vector of each first grid image is calculated to obtain grid image features. Multiple grid image features of each single-view image are combined to obtain single-view image features, where the grid division methods of the first grid images of multiple single-view images are consistent. Feature vectors are extracted for each single-view image, and multiple single-view image features are obtained accordingly. By extracting multiple single-view image features, more fusion materials can be ensured for the image surround view feature.

[0009] Next, a method for determining overlapping areas is introduced. The overlapping areas between multiple single-view images are calculated by inputting multiple angle camera parameters, including camera extrinsic parameters and camera intrinsic parameters. By determining the overlapping areas of each single-view image, it can be ensured that the fusion of each single-view image feature into the image surround view feature is correct, and the accuracy of the data pool target feature is improved.

[0010] Next, an area proportion of the overlapping region between the single-view images is calculated. First, a second grid image corresponding to the overlapping region is determined according to the overlapping region between the single-view images, and an area proportion of the second grid image between the single-view images is calculated, each second grid image corresponding to a first grid image of a plurality of single-view images having an overlapping relationship. By determining the area proportion of the overlapping region between the single-view images, the area proportion can be used as a weight, and matrix multiplication operation can be performed on the feature vectors corresponding to the grid images of each overlapping region.

[0011] Next, a method for extracting an image surround view feature is introduced. First, a first spliced feature is obtained by splicing feature vectors of second grid images having an overlapping relationship between single-view images according to an area proportion of the single-view images in the second grid images. Second, an overlapping image feature is obtained by compressing the first spliced feature based on a vector compression method, so that the vector dimension of the overlapping image feature is consistent with the vector dimension of the second grid image corresponding to the overlapping region. Finally, an image surround view feature is obtained by splicing single-view image features outside the overlapping region and overlapping image features inside the overlapping region according to the spatial relationship between the single-view images. The vector compression method includes compressing the vector of the first spliced feature through a multi-layer perception machine. By fusing image data captured by multiple camera devices as data pool data, the situation where a single camera device cannot cover the entire target object can be avoided.

[0012] Next, a method for obtaining a query image feature is introduced. The query image feature includes a plurality of single-scale image features obtained by generating a plurality of third grid images from a query image to be queried and performing feature extraction on the plurality of third grid images, respectively. First, query information of a query object is input, the query information including query text and the query image to be queried. A text semantic feature is extracted based on the query text, and a plurality of single-scale image features are extracted based on the query image to be queried, wherein the text semantic feature includes one or more keywords describing the query object. Second, the text semantic feature is fused with the plurality of single-scale image features based on the query information to obtain multi-modal fusion features of multiple scales. Finally, the multi-modal fusion features of multiple scales are fused to obtain a query feature of the query object. By fusing image data and text data as query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image can be reduced, and the query target feature of the query information can be more accurate.

[0013] Next, a single scale image feature extraction method is introduced. First, the query image is copied as a plurality of query sub-images, and each query sub-image is divided into a plurality of third grid images according to different scales, wherein the image ranges contained in the plurality of query sub-images are consistent, and the grid division methods of the third image grids of different scales are inconsistent. Second, the plurality of third grid images are generated by feature extraction to generate a plurality of single scale image features, wherein the plurality of single scale image features of the query image are independent of each other. By extracting a plurality of single scale image features from a single view image, the feature extraction of the query image can be effectively realized.

[0014] Next, a multi-modal fusion feature extraction method is introduced. First, the text semantic feature and the single scale image feature are calculated to obtain a weight factor. Second, the weight factor and the corresponding single scale image feature are multiplied to obtain a multi-modal fusion feature. Finally, the single scale image features of different scales and the text semantic features are calculated to obtain multi-modal fusion features of different scales. By extracting features of the query object at multiple different scales, the feature extraction accuracy of the query object is effectively improved.

[0015] Next, a multi-modal fusion feature fusion method is introduced. First, the plurality of multi-modal fusion features are respectively subjected to GEM transformation to obtain second spliced features by feature splicing. Second, the vector dimension of the second spliced features is reduced to the same as the vector dimension of the data pool features by a feature dimension reduction method to obtain a query feature of the query object, and the feature dimension reduction method includes principal component analysis. Before performing feature dimension reduction, the second spliced features with similar features are clustered and grouped by a clustering analysis method. By fusing image data and text data as query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image can be reduced, so that the query target feature of the query information is more accurate.

[0016] In a second aspect, a data mining system is provided, which comprises a data pool feature extraction module, a query feature extraction module, and a feature similarity determination module. The data pool feature extraction module is configured to extract a data pool feature of a target object in a data pool, wherein the data pool feature is extracted based on part or all of single-view images of the target object. The query feature extraction module is configured to extract a query feature of a query object in query information, wherein the query feature is obtained by fusing a text semantic feature and an image feature, the text semantic feature is extracted from a query text, and the query image feature is extracted from a query image. The feature similarity determination module is configured to determine a feature similarity between the data pool feature and the query feature, and output a mining result according to a threshold setting of the feature similarity. By using the data mining method provided in the present application, the query object feature of the query information can be more accurate, and the completeness of the data pool target feature can be improved.

[0017] Next, the function of the data pool feature extraction module is introduced. The data pool feature is obtained by fusing a plurality of single-view image features, which are obtained by performing feature extraction on a plurality of single-view images respectively. The plurality of single-view images include images of the target object captured at different angles. By extracting a plurality of single-view images, more fusion materials can be obtained for the image surround view feature, and the completeness of the data pool target feature can be improved. Next, the specific operation steps of the data pool feature extraction module for obtaining the data pool feature by fusing a plurality of single-view image features are introduced. First, feature extraction is performed on the plurality of single-view images respectively to obtain a plurality of single-view image features. Second, the overlapping areas between the plurality of single-view images are calculated, and the area proportions of the overlapping areas in the single-view images are calculated. Third, the plurality of single-view image features are fused to obtain an image surround view feature based on the overlapping areas between the plurality of single-view images and the area proportions of the overlapping areas in the single-view images. Finally, the image surround view feature in the target object region is taken as the data pool feature according to the position information of the target object, wherein the position information of the target object is the position information that has been labeled in the plurality of single-view images. By fusing image data captured by a plurality of cameras at different angles as data pool data, the situation that a single camera cannot cover the complete target object can be avoided.

[0018] The following describes the steps of the data pool feature extraction module to obtain multiple single-view image features by extracting features from multiple single-view images. First, each single-view image is divided into multiple first-grid images. The feature vector of each first-grid image is calculated to obtain the gridded image features. The multiple gridded image features of each single-view image are combined to form the single-view image features. The grid division method of the first-grid images of multiple single-view images is consistent. Then, feature vectors are extracted from each single-view image separately, resulting in multiple single-view image features.

[0019] The following describes the operational steps of a data pool feature extraction module for obtaining overlapping regions. By inputting parameters from multiple viewpoint cameras, the overlapping regions between multiple single-view images are calculated. These camera parameters include both extrinsic and intrinsic parameters. By determining the overlapping regions of each single-view image, it can be ensured that the fusion of features from each single-view image into the overall image view feature is accurate and consistent with the surrounding relationships, thus improving the accuracy of the target features in the data pool.

[0020] The following describes the steps of the data pool feature extraction module to determine the area ratio of overlapping regions between single-view images. First, based on the overlapping regions between single-view images, a second meshed image corresponding to the overlapping region is determined. The area ratio of the second meshed image between the single-view images is then calculated. Each second meshed image corresponds to a first meshed image of multiple overlapping single-view images. By determining the area ratio of the overlapping region between the single-view images, matrix multiplication can be performed on the feature vectors corresponding to the meshed images of each overlapping region, using the area ratio as weights.

[0021] The following describes the steps of the data pool feature extraction module to obtain image surround view features by fusing features from multiple single-view images based on the overlapping regions between them and the area proportion of the overlapping regions within the single-view images. First, based on the area proportion of the single-view images within the second gridded image, the feature vectors of the overlapping single-view images are stitched together to obtain the first stitched feature. Second, the first stitched feature is compressed using a vector compression method to obtain overlapping image features, ensuring that the vector dimension of the overlapping image features matches the vector dimension of the corresponding second gridded image within the overlapping region. Finally, based on the spatial relationship between the single-view images, the single-view image features outside the overlapping region are stitched together with the overlapping image features within the overlapping region to obtain the image surround view features. The vector compression method includes vector compression of the first stitched feature using a multilayer perceptron. By fusing image data from multiple camera devices as data pool data, the situation where a single camera device cannot cover the entire target object can be avoided.

[0022] Next, the operation of the query feature extraction module extracting the query feature is introduced. The query image feature includes a plurality of single-scale image features obtained by generating a plurality of grid images from the query image and performing feature extraction on the plurality of grid images respectively. The query feature extraction module specifically operates as follows. First, input the query information of the query object, the query information including the query text and the query image. Second, extract the text semantic feature based on the query text, and extract a plurality of single-scale image features based on the query image, wherein the text semantic feature includes one or more keywords describing the query object. Third, based on the query information, the text semantic feature is fused with the plurality of single-scale image features respectively to obtain a plurality of scale multi-modal fusion features. Finally, the plurality of scale multi-modal fusion features are fused to obtain the query feature of the query object. By fusing image data and text data as query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image can be reduced, so that the query target feature of the query information is more accurate.

[0023] Next, the operation of the query feature extraction module extracting the query feature is introduced. The query image feature includes a plurality of single-scale image features obtained by generating a plurality of grid images from the query image and performing feature extraction on the plurality of grid images respectively. The query feature extraction module specifically operates as follows. First, input the query information of the query object, the query information including the query text and the query image. Second, extract the text semantic feature based on the query text, and extract a plurality of single-scale image features based on the query image, wherein the text semantic feature includes one or more keywords describing the query object. Third, based on the query information, the text semantic feature is fused with the plurality of single-scale image features respectively to obtain a plurality of scale multi-modal fusion features. Finally, the plurality of scale multi-modal fusion features are fused to obtain the query feature of the query object. By fusing image data and text data as query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image can be reduced, so that the query target feature of the query information is more accurate.

[0024] Next, the operation of the query feature extraction module extracting the query feature is introduced. The query image feature includes a plurality of single-scale image features obtained by generating a plurality of grid images from the query image and performing feature extraction on the plurality of grid images respectively. The query feature extraction module specifically operates as follows. First, input the query information of the query object, the query information including the query text and the query image. Second, extract the text semantic feature based on the query text, and extract a plurality of single-scale image features based on the query image, wherein the text semantic feature includes one or more keywords describing the query object. Third, based on the query information, the text semantic feature is fused with the plurality of single-scale image features respectively to obtain a plurality of scale multi-modal fusion features. Finally, the plurality of scale multi-modal fusion features are fused to obtain the query feature of the query object. By fusing image data and text data as query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image can be reduced, so that the query target feature of the query information is more accurate.

[0025] Next, the operation steps of fusing the multi-modal fusion features of multiple scales in the query feature extraction module to obtain the query feature of the object to be queried are introduced. First, the GEM transformation is performed on the multiple multi-modal fusion features, and the second spliced feature is obtained by splicing the features. Then, the vector dimension of the second spliced feature is reduced to the same vector dimension as the data pool feature by the feature dimension reduction method, and the query feature of the object to be queried is obtained. The feature dimension reduction method includes principal component analysis. Before performing the feature dimension reduction, the second spliced features with similar features are clustered and grouped by the clustering analysis method. By fusing the image data and the text data as two modal data as the query information, the semantic uncertainty contained in the text and the texture uncertainty contained in the image can be reduced, so that the query information of the target feature to be queried is more accurate.

[0026] In a third aspect, a data processing device is provided, including a memory and a processor. The memory stores instructions, and the processor, when executing the instructions stored in the memory, can implement the method of any embodiment of the first aspect.

[0027] In a fourth aspect, a non-volatile storage medium is provided, including a plurality of instructions, which, when executed by a data processing device, can implement the method required by any embodiment of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings required to be used in the embodiments of the present application or the background art will be described below.

[0029] Figure 1 is a schematic diagram of a vehicle-mounted camera device recording an environment image provided by the present application;

[0030] Figure 2 is a schematic diagram of the architecture of a data mining system provided by the present application;

[0031] Figure 3 is a schematic diagram of the architecture of a data pool feature extraction module provided by the present application;

[0032] Figure 4 is a schematic diagram of the overlapping region of two images based on different viewing angles provided by the present application;

[0033] Figure 5 is a schematic diagram of obtaining image surround view features by fusing multiple single-view image features provided by the present application;

[0034] Figure 6 is a schematic diagram of the architecture of a query feature extraction module provided by the present application;

[0035] Figure 7is a schematic diagram of multi-modal feature fusion of single-scale image features provided by the present application;

[0036] Figure 8 is a schematic diagram of multi-scale feature fusion of multi-modal fusion features of different scales provided by the present application;

[0037] Figure 9 is a schematic diagram of the overall flow of a data mining method provided by the present application;

[0038] Figure 10 is a schematic diagram of the flow of data pool feature extraction provided by the present application;

[0039] Figure 11 is a schematic diagram of the flow of query feature extraction provided by the present application;

[0040] Figure 12 is a schematic diagram of a data mining device provided by the present application;

[0041] Figure 13 is a schematic diagram of the structure of a data processing device provided by the present application. DETAILED DESCRIPTION

[0042] For the purposes of the present application, the technical solutions and advantages will be more apparent, the following will be described by embodiments of the present application, clear and complete description of the technical solutions of the present application, obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without doing creative work are within the scope of the present application.

[0043] In the description of the present application, unless otherwise explicitly defined, the setting of the words such as words should be broad understanding, the person skilled in the art can combine the specific content of the technical solution to reasonably determine the specific meaning of the above words in the present application.

[0044] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific embodiment", or "some examples" means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0045] For the convenience of the relevant personnel to understand the present application, the application scenarios suitable for the present application are briefly described by combining the embodiments and the drawings.

[0046] Please refer to Figure 1 , Figure 1 A schematic diagram of recording environmental images by a vehicle-mounted camera device is provided in the present application. In the automatic driving business, a vehicle can record images taken by multi-view vehicle-mounted camera devices as objects for judging the surrounding environment of the vehicle. As shown in the vehicle Figure 1 The vehicle is recorded by six vehicle-mounted camera devices 110 with different views, and each vehicle-mounted camera device 110 generates corresponding image data 120. In the data mining scheme in the field of automatic driving, the image data 120 taken by the vehicle-mounted camera device 110 is usually uploaded to the data pool 130. By inputting the data in the data pool 130 into the data mining tool 150, the data mining tool 150 extracts the data mining result 160 by querying the information 140 in the data pool 130 and achieving data mining on the target to be queried. Among them, the data pool 130 is the collected data stored in the cloud, and the query information 140 is the input information of the user under the current mining task, which is used to represent the data mining result 160 containing the target to be queried in the data pool 130.

[0047] In the current data mining scheme, the data form in the data pool 130 is mainly obtained according to a single image data 120, and each image data 120 corresponds to a single picture taken by each vehicle-mounted camera device 110, and the data mining tool 150 extracts corresponding image features according to the single picture. Under normal circumstances, since the range of a single picture taken by each vehicle-mounted camera device 110 is limited, when the target to be queried appears at the edge of a single image data 120, the image features that can be extracted by the data mining tool 150 will not match the query information 140.

[0048] The data form of the above data pool 130 uses the existing technology of mining by using single picture information, which may only be able to capture part of the content of the target (such as the head or tail of the object to be queried) for a larger object, and it is difficult to represent the complete features of the target to be queried.

[0049] In the current data mining scheme, the query information 140 mainly has two forms, namely text information-based query and image information-based query. The first type is a data retrieval scheme based on query text, that is, the input query information is a text statement, such as "a truck on the road" and the like. In this scheme, there are mainly two parts of image feature extraction and text feature extraction. The text feature extraction encodes the query statement into a text feature, and the image feature extraction encodes the images in the data pool into image features, and outputs the data samples with higher similarity. The second type is a data retrieval scheme based on query image, that is, the input query information is an image sample. In this scheme, the image feature extraction module is mainly used to extract the image features of the data pool and the features of the query image at the same time, and output the data samples with higher similarity.

[0050] The above two forms of query information 140 mainly have the following two problems. First, only simple text statement description is supported, and the retrieval performance for target objects that need to be described in detail is poor, such as "a flatbed trailer carrying an excavator" and the like. Second, for some complex environmental features, it is difficult to accurately describe them directly with text, such as "in the dark environment, the headlight of the rear vehicle causes glare to the front vehicle" and the like. The prior art only performs data mining by inputting a single type of query information (text or image). As can be seen from the above analysis, text query information focuses more on the high-level semantic information of the target to be mined, while image query information contains more detailed texture information of the target. A single type of query information is difficult to provide accurate query target features.

[0051] As Figure 1The embodiments shown are only as an example, and in other possible embodiments, the number of vehicle-mounted camera devices for recording road test data can be more or less, the position distribution of the vehicle-mounted camera devices on the vehicle, and the camera device angle direction can be customized and adjusted, which is not limited here. The embodiments shown in this application are only a specific application example of the data mining method, and in other possible embodiments, the vehicle-mounted camera device 110 for collecting data for data mining can not only be set on a car or a car, but also can be set on a land traveling traffic vehicle and tool equipment including a train, a truck, an electric vehicle, a bicycle, a skateboard, etc., and an air flying traffic vehicle and tool equipment including an aircraft, a drone, a weather observer, a helicopter, a spacecraft, etc., and a sea surface or ocean traveling traffic vehicle and tool equipment including a ship, a yacht, a submarine, etc., and a wearable device set on a human body such as a helmet and a monitoring camera device; the image data 120 includes not only optical image signals captured by optical camera devices, but also photoelectric digital signals collected by laser radar and millimeter wave radar; the data pool 130 includes not only the storage resources of the cloud server, but also non-cloud physical storage resources; the form of the query information 140 includes not only text information and image information, but also voice information, digital signals, etc.

[0052] To solve the problems of the prior art in the embodiments shown, Figure 1 The present application provides a data mining system, method and device based on joint text and image information. The data mining system, method and device provided by the present application can reduce the semantic uncertainty contained in the text and the texture uncertainty contained in the image by fusing image data and text data as query information, so that the query target features of the query information are more accurate. At the same time, by fusing image data captured by multiple angle camera devices as data pool data, the situation that a single camera device cannot cover the complete target object can be avoided, and the completeness of the target features of the data pool can be improved.

[0053] Please refer to Figure 2 , Figure 2 The architecture diagram of a data mining system provided by the present application is shown. As shown in Figure 2 The data mining system 200 provided by the present application includes a data pool feature extraction module 210, a query feature extraction module 220, and a feature similarity determination module 230.

[0054] Firstly, the data pool feature extraction module 210 is introduced, which is used to extract the data pool feature 203 of the target object in the data pool 201. The data pool feature 203 is extracted based on part or all of the single-view images of the target object. The data pool feature extraction module 210 obtains the data pool feature 203 by fusing multiple single-view image features, and the multiple single-view image features are obtained by respectively extracting features from multiple single-view images, and the multiple single-view images include images of the target object captured at different angles.

[0055] For the convenience of understanding, the function of the data pool feature extraction module 210 is described in detail through an embodiment.

[0056] Please refer to Figure 3 , Figure 3 The architecture diagram of the data pool feature extraction module provided in the present application is shown. The data pool feature extraction module 210 shown in the present application includes a single-view image feature extraction module 310 and a surround-view feature fusion module 320. It can be understood that Figure 3 The data pool data 301 shown corresponds to the data pool 201 shown in Figure 2 , and also corresponds to the image data 120 captured by the six vehicle-mounted camera devices 110 shown in Figure 1 .

[0057] As shown in Figure 3 , the multiple single-view image features are obtained by respectively extracting features from the multiple single-view images in the data pool data 301 through the single-view image feature extraction module 310. In the embodiment shown in Figure 3 , the data pool data 301 contains six single-view images in different directions, which are the front, back, left front, right front, left back, and right back directions. Firstly, the single-view image feature extraction module 310 extracts multiple single-view images in different angles at the same time stamp according to the time stamp of each image data in the data pool, and divides each single-view image at the same time stamp into multiple MxN distributed first grid images. Secondly, the feature vector of each first grid image is calculated to obtain the grid image feature. Thirdly, the multiple grid image features of each single-view image are combined to obtain the single-view image feature 302, wherein the grid division manners of the first grid images of the multiple single-view images are consistent. Finally, the feature vector of each single-view image is extracted to obtain the multiple single-view image features 302, which include the front feature, back feature, left front feature, right front feature, left back feature, and right back feature in this embodiment.

[0058] Subsequently, the overlap regions between the plurality of single-view images are calculated by the surround-view feature fusion module 320, and the area proportions of the overlap regions between the single-view images are calculated. The surround-view feature fusion module 320 also fuses the plurality of single-view image features to obtain image surround-view features based on the overlap regions between the plurality of single-view images and the area proportions of the overlap regions between the single-view images, and takes the image surround-view features in the target object region as the data pool features 304 according to the position information of the target object, wherein the position information of the target object is the position information that has been labeled in the plurality of single-view images.

[0059] It can be understood that, Figure 3 The embodiments of the vehicle-mounted camera device are only an example, and in other possible embodiments, the number of device view angles of the vehicle-mounted camera device can be more or less, and the arrangement of the device view angles can be arbitrary, which is not limited here.

[0060] For the convenience of understanding of the relevant personnel, more detailed descriptions of the single-view image feature extraction module 310 and the surround-view feature fusion module 320 are described in Figure 4 and Figure 5 the embodiments of the application.

[0061] Please refer to Figure 4 , Figure 4 is an overlap region diagram of two images based on different view angles provided by the application. As Figure 4 indicated, the two view images are a first single-view image 401A and a second single-view image 402A, and the overlap region between the first single-view image 401A and the second single-view image 402A is determined according to the two camera device parameters for shooting the first single-view image 401A and the second single-view image 402A. The camera device parameters include camera device extrinsic parameters and camera device intrinsic parameters. The camera device extrinsic parameters refer to the position and direction parameters of the camera relative to the scene being shot, which are usually described by Euler angles or rotation matrix (r) and translation vector (t). These parameters describe the position and direction relationship of the camera in three-dimensional space, and can be used to calculate the actual position and size of the object in the image. By adjusting the extrinsic parameters, the shooting view angle can be changed, and factors such as image size, shape and placement angle are affected. The camera device intrinsic parameters refer to the inherent parameters of the camera itself, including focal length, principal point coordinates, distortion coefficient, etc. These parameters remain unchanged when shooting images, and play a very important role in imaging quality and measurement accuracy. For example, the larger the focal length, the clearer and more detailed the image; the smaller the distortion coefficient, the closer the image to the actual object shape.

[0062] As Figure 4As shown, the first single-view image 401A is divided into a grid by the single-view image feature extraction module 310 to obtain the first gridded image 401B of the first single-view image 401A, and the second single-view image 402A is divided into a grid to obtain the first gridded image 402B of the second single-view image 402A. According to the camera device parameters, the overlapping area of ​​the gridded images 401B and 402B is determined by the surround view feature fusion module 320. The overlapping area corresponds to the second gridded image 403B, and the area ratio of each grid in the second gridded image 403B between the single-view images is calculated. Typically, each second gridded image 403B corresponds to the first gridded image of multiple overlapping single-view images.

[0063] Understandable. Figure 4 The embodiment shown is only an example. In other possible embodiments, the operation of meshing the first single-view image 401A and the second single-view image 402A can also be performed before determining the overlapping area 403A of the two, which is not limited here.

[0064] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating how to obtain image surround view features by fusing multiple single-view image features, as provided in this application.

[0065] like Figure 5 The illustrated embodiment includes multiple single-view image features 501 and an image surround view feature 502 that fuses the multiple single-view image features 501. First, after the surround view feature fusion module 320 has determined the overlapping regions corresponding to each single-view image and obtained a second gridded image of the overlapping region, the area ratio between the single-view images in the second gridded image is extracted according to the camera device parameters. The feature vectors of the second gridded images with overlapping relationships among the multiple single-view image features 501 are then stitched together to obtain a first stitched feature. Second, the first stitched feature is compressed using a vector compression method to obtain overlapping image features, making the vector dimension of the overlapping image features consistent with the vector dimension of the corresponding second gridded image of the overlapping region. Finally, based on the spatial relationship between the single-view images, the single-view image features outside the overlapping region are stitched together with the overlapping image features within the overlapping region to obtain the image surround view feature 502. The vector compression method includes compressing the feature vector of the first stitched feature using a multilayer perceptron. It can be understood that... Figure 5 The illustrations provided are merely concrete examples for ease of understanding. In actual computation, each of the multiple single-view image features 501 and the image panoramic feature 502 is composed of multiple gridded feature vectors, not image data or... Figure 5in the form of a specific legend.

[0066] Next, the query feature extraction module 220 is introduced, which is used to extract the query feature 204 of the query object in the query information 202. The query feature 204 is obtained by fusing the text semantic feature and the image feature. The text semantic feature is extracted from the query text, which describes the query object. The image feature is extracted from the query image, which includes part or all of the query object. The image feature includes multiple single-scale image features obtained by generating multiple grid images from the query image and extracting features from the multiple grid images.

[0067] For ease of understanding, the function of the query feature extraction module 220 is described in detail through an embodiment.

[0068] Please refer to Figure 6 , Figure 6 The architecture of a query feature extraction module provided by the present application is shown. The query feature extraction module 220 shown in the embodiment includes a text semantic feature extraction module 610, an image multi-scale feature extraction module 620, a multi-modal feature fusion module 630, and a multi-scale feature fusion module 640. For the convenience of understanding of those skilled in the related art, more detailed descriptions of the multi-modal feature fusion module 630 and the multi-scale feature fusion module 640 are described in the embodiments of Figure 7 and Figure 8 .

[0069] As shown in Figure 6 , the query feature extraction module 220 inputs the query information of the query object, which includes the query text 601 and the query image 602.

[0070] The text semantic feature extraction module 610 extracts the text semantic feature 603 based on the query text 601, wherein the text semantic feature 603 includes one or more keywords describing the query object. At the same time, the image multi-scale feature extraction module 620 extracts multiple single-scale image features 604 based on the query image 602. The specific steps include copying the query image 602 into multiple query sub-images, dividing each query sub-image into multiple third grid images according to different scales, and the image ranges contained in the multiple query sub-images are consistent, but the grid division methods of the third image grids of different scales are inconsistent. The multiple third grid images are generated into multiple single-scale image features 604 through feature extraction, and the multiple single-scale image features 604 of the query image 602 are independent of each other.

[0071] In the embodiment of the present application, the image multi-scale feature extraction module 620 obtains three independent single-scale image features 604 corresponding to the third grid image of three different scales by dividing the query image 602, which are single-scale image feature (large scale), single-scale image feature (medium scale) and single-scale image feature (small scale) respectively. In other possible embodiments, the number of generated single-scale image features 604 can also be less or more, which is not limited here.

[0072] Please refer to Figure 7 , Figure 7 for a schematic diagram of a single-scale image feature provided by the present application for multi-modal feature fusion. The multi-modal feature fusion module 630 fuses the text semantic feature 603 with the plurality of single-scale image features 604 based on the query information to obtain a plurality of multi-modal fusion features 605 of different scales.

[0073] As shown in Figure 7 , the embodiment introduces the specific operation of the multi-modal feature fusion module 630 for fusing the text semantic feature and the single-scale image feature to obtain a multi-modal fusion feature of a certain scale. First, the feature similarity calculation 710 is performed on the text semantic feature 701 and the single-scale image feature 702 to obtain a weight factor 703. Then, the vector product operation 720 is performed on the weight factor 703 and the corresponding single-scale image feature 702 to obtain a multi-modal fusion feature 704.

[0074] According to the number of single-scale image features 604 extracted by the image multi-scale feature extraction module 620, the operation of obtaining the multi-modal fusion feature 704 as shown in Figure 7 is repeated by performing the feature similarity calculation 710 and the vector product operation 720 on the single-scale image features 702 of different scales and the text semantic feature 701 to obtain multi-modal fusion features 704 of multiple scales respectively.

[0075] Please refer to Figure 8 , Figure 8 for a schematic diagram of a multi-scale feature fusion of different scale multi-modal fusion features provided by the present application. The multi-scale feature fusion module 640 fuses the multi-scale multi-modal fusion features 605 to obtain the query feature 606 of the query object.

[0076] As shown in Figure 8As shown, the embodiment introduces the specific operation of the multi-scale feature fusion module 640 fusing multi-modal fusion features of different scales to obtain multi-scale fusion features. First, after the GEM transformation 810 is performed on the plurality of multi-modal fusion features 801 respectively, the second spliced features 802 are obtained through feature splicing 820. Subsequently, the vector dimension of the second spliced features 802 is reduced to the same vector dimension as the data pool features through the feature dimension reduction method 830 to obtain the query features 803 of the object to be queried, and the feature dimension reduction method includes principal component analysis. In particular, in using the feature dimension reduction method, the second spliced features 802 with similar features are clustered and grouped through the cluster analysis method.

[0077] Finally, the feature similarity determination module 230 is introduced. This module is used to calculate the feature similarity between the data pool features 203 and the query features 204 obtained by the data pool feature extraction module 210 and the query feature extraction module 220 respectively according to the above data pool feature extraction module 210 and the query feature extraction module 220, and output the mining results 205 related to the object to be queried among the target objects through the threshold setting of the feature similarity. The precision of data mining is described by the feature similarity in this application. The feature similarity is a real number with a numerical range of 0-1. The higher the numerical value of the feature similarity, the closer the features of the target object to the features of the object to be queried. When the numerical value of the feature similarity is greater than the set threshold value, it is determined that the target object in the data pool picture has similar features to the object to be queried, and is used as the mining result that can be output.

[0078] Through the embodiment as shown in Figures 2 to 8 , the data mining system based on the combination of image and text information provided by the present application is introduced. Next, combined with the above-mentioned data mining system, a data mining method based on the combination of image and text information provided by the present application is introduced through specific operation steps.

[0079] Please refer to Figure 9 , Figure 9 for the overall flowchart of the data mining method provided by the present application. The data mining method based on the combination of image and text information shown in the embodiment can be used to realize the data mining system based on the combination of image and text information as shown in Figure 2 . The specific steps are as follows:

[0080] S101: Extract the data pool features of the target object in the data pool.

[0081] From the data pool, first need to extract single perspective image from the data pool, and then fuse multiple single perspective image features to obtain image surround view features, and finally determine the data pool features of the target object through the labeled position information of the target object in the single perspective image. Among them, multiple single perspective image features are obtained by respectively extracting features from multiple single perspective images, and the multiple single perspective images include images obtained by collecting the target object under multiple different perspectives.

[0082] For the convenience of understanding, the specific steps of data pool feature extraction are introduced in detail through the following embodiments. Please refer to Figure 10 , Figure 10 A flowchart of data pool feature extraction provided by the present application is shown. The specific operation steps are as follows:

[0083] S201: Extracting features from multiple single perspective images to obtain multiple single perspective image features.

[0084] The premise of fusing multiple single perspective images is that each single perspective image has the same timestamp. Each single perspective image is divided into multiple first grid images, the feature vector of each first grid image is calculated to obtain a grid image feature, and multiple grid image features of each single perspective image are combined into a single perspective image feature, wherein the grid division method of the first grid image of multiple single perspective images is consistent. Extracting feature vectors from each single perspective image respectively, and corresponding multiple single perspective image features are obtained.

[0085] S202: Calculate the overlapping area between multiple single perspective images, and calculate the area proportion of the overlapping area between single perspective images.

[0086] The overlapping area between multiple single perspective images is calculated by inputting the parameters of multiple perspective camera devices, including camera device external parameters and camera device internal parameters. According to the overlapping area between single perspective images, the second grid image corresponding to the overlapping area is determined, and the area proportion of the second grid image between single perspective images is calculated respectively, each second grid image corresponds to the first grid image of multiple single perspective images with overlapping relationship.

[0087] S203: Based on the overlapping area between multiple single perspective images and the area proportion of the overlapping area between single perspective images, fuse multiple single perspective image features to obtain image surround view features.

[0088] The feature vectors of the second meshed images having an overlapping relationship between the single-view images are spliced according to the area proportion of the single-view images in the second meshed images, to obtain first spliced features. The first spliced features are compressed based on a vector compression method to obtain overlapping image features, so that the vector dimension of the overlapping image features is consistent with the vector dimension of the second meshed images of the corresponding overlapping region. The single-view image features outside the overlapping region and the overlapping image features inside the overlapping region are spliced according to the spatial relationship between the single-view images, to obtain image surround-view features. The vector compression method includes vector compression of the first spliced features by a multilayer perception machine.

[0089] For the calculation method of the overlapping region, please refer to the related description in the embodiment shown in Figure 4 , which will not be repeated here. For the calculation method of the image surround-view features, please refer to the related description in the embodiment shown in Figure 5 , which will not be repeated here.

[0090] S204: According to the position information of the target object, the image surround-view features in the target object region are taken as data pool features.

[0091] The position information of the target object has been labeled in the data pool, and the corresponding position information of the target object is included when the single-view images are extracted. At the same time, each single-view image has included the corresponding time stamp from the data pool.

[0092] Through the related operations of S201-S204, the target of data pool feature extraction shown in S101 can be achieved.

[0093] S102: Extract the query features of the object to be queried in the query information.

[0094] The query image features include a plurality of single-scale image features obtained by generating a plurality of meshed images from the query image and performing feature extraction on the plurality of meshed images, respectively.

[0095] For ease of understanding, the specific steps of data pool feature extraction will be described in detail through an embodiment. Please refer to Figure 11 , Figure 11 for a flowchart of query feature extraction provided by the present application. The specific operation steps are as follows:

[0096] S301: Input the query information of the object to be queried, which includes the query text and the query image.

[0097] The query text records the text used to describe the object to be queried, and the query image includes part or all of the object to be queried.

[0098] S302: Extract text semantic features based on the text to be queried, and extract a plurality of single-scale image features based on the image to be queried.

[0099] Firstly, the text semantic features include one or more keywords describing the object to be queried. Secondly, the image to be queried is copied into a plurality of sub-images to be queried, and each sub-image to be queried is divided into a plurality of third grid images according to different scales, wherein the image ranges contained in the plurality of sub-images to be queried are consistent, and the grid division manners of the third image grids of different scales are inconsistent. Finally, the plurality of third grid images are generated into a plurality of single-scale image features through feature extraction, wherein the plurality of single-scale image features of the image to be queried are independent of each other.

[0100] S303: Based on the query information, the text semantic features are respectively fused with the plurality of single-scale image features to obtain a plurality of scale multi-modal fusion features.

[0101] Firstly, the feature similarity calculation is performed on the text semantic features and the single-scale image features to obtain a weight factor. Secondly, the vector product operation is performed on the weight factor and the corresponding single-scale image feature to obtain the multi-modal fusion feature. Finally, the feature similarity calculation and the vector product operation are performed on the single-scale image features of different scales and the text semantic features to obtain a plurality of scale multi-modal fusion features.

[0102] The specific operation of the multi-modal feature fusion can be combined with the related description in the embodiment shown in Figure 7 The specific operation of the multi-modal feature fusion can be combined with the related description in the embodiment shown in Figure 8 .

[0103] S304: Fuse the plurality of scale multi-modal fusion features to obtain the query feature of the object to be queried.

[0104] Firstly, the GEM transformation is performed on the plurality of multi-modal fusion features respectively, and the second splicing feature is obtained through feature splicing. Subsequently, the vector dimension of the second splicing feature is reduced to the same as the vector dimension of the data pool feature through the feature dimension reduction method to obtain the query feature of the object to be queried, and the feature dimension reduction method includes the principal component analysis method.

[0105] S103: Determine the feature similarity of the data pool feature and the query feature, and output the mining result according to the threshold setting of the feature similarity.

[0106] In order to save space, the related concepts and descriptions of the data mining method in the embodiments of the present application are referred to, such as Figures 2 to 8For the related introduction of a data mining system provided in the present application, no redundant description is made. The data mining method shown in the present application can reduce the semantic uncertainty contained in the text and the texture uncertainty contained in the image by fusing the image data and the text data as query information, so that the to-be-queried target feature of the query information is more accurate. At the same time, by fusing the image data captured by multiple perspective camera devices as the data pool data, the situation that a single camera device cannot cover the complete target object can be avoided, and the integrity of the data pool target feature is improved.

[0107] Please refer to Figure 12 , Figure 12 The schematic diagram of a data mining device provided in the present application is shown. The data mining device shown in the present application comprises a single-perspective image feature extraction module 1210, a surround-view feature fusion module 1220, a text semantic feature extraction module 1230, an image multi-scale feature extraction module 1240, a multi-modal feature fusion module 1250, a multi-scale feature fusion module 1260, and a feature similarity determination module 1270.

[0108] The single-perspective image feature extraction module 1210 is configured to perform feature extraction on multiple single-perspective images in the data pool to obtain multiple single-perspective image features. First, according to the time stamps of the image data in the data pool, multiple single-perspective images of different perspectives at the same time stamp are extracted, and each single-perspective image at the same time stamp is divided into multiple MxN distributed first grid images. Second, the feature vector of each first grid image is calculated to obtain a grid image feature. Third, multiple grid image features of each single-perspective image are combined into a single-perspective image feature, wherein the grid division manners of the first grid images of multiple single-perspective images are consistent. Finally, the feature vector of each single-perspective image is extracted to obtain multiple single-perspective image features.

[0109] The surround-view feature fusion module 1220 is configured to calculate the overlapping areas between multiple single-perspective images and the area proportions of the overlapping areas between the single-perspective images. The surround-view feature fusion module is further configured to fuse multiple single-perspective image features to obtain image surround-view features based on the overlapping areas between multiple single-perspective images and the area proportions of the overlapping areas between the single-perspective images, and to take the image surround-view features in the target object region as the data pool features according to the position information of the target object, wherein the position information of the target object is the position information that has been labeled in multiple single-perspective images. For the calculation method of the overlapping areas, please refer to the related description in the embodiment shown in Figure 4 For the calculation method of the image surround-view features, please refer to the related description in the embodiment shown in Figure 5 ,

[0110] In other possible embodiments, the monocular image feature extraction module 1210 and the surround view feature fusion module 1220 can also be combined into a data pool feature extraction module capable of extracting image surround view features from data pool data.

[0111] The text semantic feature extraction module 1230 is configured to extract text semantic features based on the query text, wherein the text semantic features include one or more keywords describing the query object.

[0112] The image multi-scale feature extraction module 1240 is configured to extract a plurality of single-scale image features based on the query image. The specific steps include copying the query image into a plurality of query sub-images, dividing each query sub-image into a plurality of third grid images according to different scales, the image ranges contained in the plurality of query sub-images being consistent, and the grid division manners of the third image grids of different scales being inconsistent. The plurality of third grid images are subjected to feature extraction to generate a plurality of single-scale image features, and the plurality of single-scale image features of the query image are independent of each other.

[0113] The multi-modal feature fusion module 1250 is configured to fuse the text semantic features 603 with the plurality of single-scale image features based on the query information to obtain a plurality of multi-modal fusion features of different scales. First, the text semantic features 701 and the single-scale image features are subjected to feature similarity calculation to obtain weight factors. Subsequently, the weight factors and the corresponding single-scale image features are subjected to vector product operation to obtain multi-modal fusion features. Finally, the single-scale image features of different scales and the text semantic features are subjected to feature similarity calculation and vector product operation to obtain multi-modal fusion features of a plurality of scales, respectively.

[0114] The multi-scale feature fusion module 1260 is configured to fuse the multi-modal fusion features of a plurality of scales to obtain query features of the query object. First, the plurality of multi-modal fusion features are subjected to GEM transformation, and second splicing features are obtained through feature splicing. Subsequently, the vector dimension of the second splicing features is reduced to the same as that of the data pool features through a feature dimension reduction method to obtain the query features of the query object, and the feature dimension reduction method includes principal component analysis. In particular, before using the feature dimension reduction method, the second splicing features with similar features are clustered and grouped through a clustering analysis method.

[0115] In other possible embodiments, the image multi-scale feature extraction module 1240, the multi-modal feature fusion module 1250, and the multi-scale feature fusion module 1260 can also be combined into a query feature extraction module capable of extracting multi-modal multi-scale query features from query information.

[0116] The feature similarity determination module 1270 is configured to calculate the feature similarity between the data pool features and the query features obtained above, and output the mining result related to the object to be queried among the target objects through a threshold setting of the feature similarity. The feature similarity is used to describe the accuracy of data mining in the present application, and is a real number with a numerical range of 0-1. The higher the numerical value of the feature similarity, the closer the features of the target object to the features of the object to be queried. When the numerical value of the feature similarity is greater than the set threshold, it is determined that the target object in the data pool picture has similar features to the object to be queried, and is used as the mining result that can be output.

[0117] More related concepts and specific descriptions of data mining are described in the data mining system and the data mining method as shown in Figures 2 to 11 for the sake of brevity.

[0118] Referring to Figure 13 , Figure 13 is a structural schematic diagram of a data processing device provided by the present application. The data processing device shown in the present application comprises one or more first processors 1310, a data collector 1320, a communication interface 1330, an external memory 1340 and a bus 1350. The first processor 1310, the data collector 1320, the communication interface 1330 and the external memory 1340 can be connected through the bus 1350.

[0119] The data collector 1320 is configured to collect the related data generated in the construction process of the data mining method as described in S101-S103, and acquire data between the modules.

[0120] The first processor 1310 comprises one or more general-purpose processors, wherein the general-purpose processor can be any type of device capable of processing electronic instructions, including a central processing unit (CPU), a microprocessor, a microcontroller, a main processor, a controller and an ASIC (application specific integrated circuit), etc. The first processor 1310 executes various types of digital storage instructions, such as software or firmware programs stored in the external memory 1340, which enables the data processing device to provide a wide variety of services. For example, the first processor 1310 can execute programs or process data to perform at least part of the methods discussed herein.

[0121] The communication interface 1330 can be a wired interface (e.g., an Ethernet interface) for communicating with other data processing devices or users, for enabling, and the like, as described in detail above.

[0122] The external memory 1340 can include a volatile memory (e.g., a random access memory (RAM)), and can also include a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD)). The external memory 1340 can also include a combination of the foregoing kinds of memories. The external memory 1340 can store data and / or program codes.

[0123] The first processor 1310 can execute the contents described in steps S101-S103, steps S201-S204, and steps S301-S304 by invoking the program codes stored in the external memory 1340. For the sake of brevity, the detailed description of the present embodiment can be specifically referred to Figures 2 to 11 The related descriptions and specific contents of the data mining system and the data mining method.

[0124] Although the present disclosure describes functions performed by certain virtual components and physical entities, in a cloud service system application scenario, some or all of the above-mentioned coding relationship management processes can be distributed among multiple virtual components and entities, and one or more instances of the processes can be executed on a distributed system, a virtual machine, and one or more cloud service systems.

[0125] The present disclosure provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements all the steps of the above-mentioned target reporting method, for example, the processor executes the computer program to implement the processes and related operations described in steps S101-S103, steps S201-S204, and steps S301-S304.

[0126] The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device to perform all or part of the steps of the method described in the embodiments of the present application. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server, cloud service center or data center to another website, computer, server, cloud service center or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.).

[0127] The computer-readable storage medium can be any available medium accessible by the computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets, and can also be a cloud storage resource provided by a cloud server. The available medium can be a magnetic medium (for example, a floppy disk, a storage disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state storage disk, SSD), etc.

[0128] In addition, in Figure 13 The logical instructions in the external memory 1340 shown can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or part of the technical solutions can be embodied in the form of a software product.

[0129] The present application also provides a computer program product, which includes a computer program that is executed by a processor to implement all the steps of the above-mentioned target reporting method, for example, the processor executes the computer program to implement the processes of steps S101-S103, steps S201-S204, steps S301-S304 and related operations.

[0130] The above-described embodiments are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0131] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones. The modification or replacement does not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data mining method, characterized in that, include: Multiple single-view image features are obtained by extracting features from multiple single-view images respectively; Based on the overlapping regions between the multiple single-view images and the area ratio of the overlapping regions between the single-view images, the features of multiple single-view images are fused to obtain the image surround view feature. Based on the location information of the target object, the image surround view features within the target object region are used as data pool features, wherein the location information of the target object is the location information that has been marked in the multiple single-view images. Extract query features of the object to be queried from the query information. The query features are obtained by fusing text semantic features and query image features. The text semantic features are extracted from the text to be queried, which contains text describing the object to be queried. The query image features are extracted from the image to be queried, which includes part or all of the object to be queried. Determine the feature similarity between the data pool features and the query features, and output the mining results based on the threshold set for the feature similarity.

2. The method according to claim 1, characterized in that, The multiple single-view images include images of the target object acquired from multiple different viewpoints.

3. The method according to claim 1, characterized in that, The step of extracting features from multiple single-view images to obtain multiple single-view image features includes: Each single-view image is divided into multiple first gridded images. The feature vector of each first gridded image is calculated to obtain gridded image features. The multiple gridded image features of each single-view image are combined into single-view image features. The grid division method of the first gridded images of the multiple single-view images is consistent. Feature vectors are extracted from each of the single-view images to obtain multiple single-view image features.

4. The method according to claim 3, characterized in that, The overlapping regions between the multiple single-view images are obtained through calculation, including: The overlapping region between the multiple single-view images is calculated by inputting parameters from multiple viewpoint camera devices, wherein the camera device parameters include extrinsic parameters and intrinsic parameters of the camera devices.

5. The method according to claim 4, characterized in that, The proportion of the overlapping region between single-view images is calculated, including: Based on the overlapping area between the single-view images, a second meshed image corresponding to the overlapping area is determined, and the area ratio occupied by the second meshed image between the single-view images is calculated respectively. Each second meshed image corresponds to the first meshed image of multiple overlapping single-view images.

6. The method according to claim 5, characterized in that, The image surround view feature is obtained by fusing multiple single-view image features based on the overlapping regions between the multiple single-view images and the area ratio of the overlapping regions between the single-view images, including: Based on the area ratio between single-view images in the second gridded image, the feature vectors of the second gridded images that have overlapping relationships between the single-view images are stitched together to obtain the first stitching feature. The first stitching feature is compressed using a vector compression method to obtain overlapping image features, so that the vector dimension of the overlapping image features is consistent with the vector dimension of the second meshed image of the corresponding overlapping region. Based on the spatial relationship between the single-view images, the single-view image features outside the overlapping region and the overlapping image features within the overlapping region are stitched together to obtain the image panoramic features.

7. The method according to claim 6, characterized in that, The vector compression method includes using a multilayer perceptron to compress the vector of the first spliced ​​feature.

8. The method according to any one of claims 1 to 7, characterized in that, The query image features include multiple single-scale image features obtained by generating multiple third-grid images based on the query image and extracting features from the multiple third-grid images respectively.

9. The method according to claim 8, characterized in that, The extraction of query features of the object to be queried from the query information includes: Input the query information for the object to be queried, which includes the text and image to be queried; The semantic features of the text are extracted based on the text to be queried, and the multiple single-scale image features are extracted based on the image to be queried, wherein the semantic features of the text include one or more keywords describing the object to be queried; Based on the query information, the text semantic features are fused with the multiple single-scale image features to obtain multi-modal fusion features at multiple scales; By fusing the multimodal fusion features at multiple scales, the query features of the object to be queried are obtained.

10. The method according to claim 9, characterized in that, The step of extracting the multiple single-scale image features based on the image to be queried includes: The image to be queried is copied into multiple sub-images to be queried, and each sub-image to be queried is divided into multiple third gridded images according to different scales. The image range contained in the multiple sub-images to be queried is consistent, and the grid division method between the third gridded images at different scales is inconsistent. Multiple single-scale image features are generated by feature extraction from multiple third-mesh images, wherein the multiple single-scale image features of the query image are independent of each other.

11. The method according to claim 10, characterized in that, The step of fusing the text semantic features with corresponding single-scale image features to obtain multi-scale multimodal fusion features includes: The text semantic features and the single-scale image features are used to calculate feature similarity to obtain a weighting factor; The weighting factors and the corresponding single-scale image features are multiplied by a vector to obtain multimodal fusion features; By performing feature similarity calculation and vector multiplication on the single-scale image features at different scales and the text semantic features, the multimodal fusion features at multiple scales are obtained respectively.

12. The method according to claim 11, characterized in that, The process of fusing the multimodal fusion features at multiple scales to obtain the query features of the object to be queried includes: GEM transformation is performed on each of the multiple multimodal fusion features, and the second concatenated feature is obtained by feature concatenation. The vector dimension of the second concatenated feature is reduced to the same as the vector dimension of the data pool feature by the feature dimensionality reduction method, thereby obtaining the query feature of the object to be queried. The feature dimensionality reduction method includes principal component analysis.

13. The method according to claim 12, characterized in that, Before performing the feature dimensionality reduction, the second spliced ​​features with similar characteristics are clustered together using cluster analysis.

14. A data mining system, characterized in that, include: The data pool feature extraction module is used to extract features from multiple single-view images to obtain multiple single-view image features; Based on the overlapping regions between the multiple single-view images and the area ratio of the overlapping regions between the single-view images, the features of multiple single-view images are fused to obtain the image surround view feature. Based on the location information of the target object, the image surround view features within the target object region are used as data pool features, wherein the location information of the target object is the location information that has been marked in the multiple single-view images. The query feature extraction module is used to extract query features of the object to be queried from the query information. The query features are obtained by fusing text semantic features and query image features. The text semantic features are extracted from the text to be queried, which contains text describing the object to be queried. The query image features are extracted from the image to be queried, which includes part or all of the object to be queried. The feature similarity determination module is used to determine the feature similarity between the data pool features and the query features, and output the mining results according to the threshold setting of the feature similarity.

15. The system according to claim 14, characterized in that, The multiple single-view images include images of the target object acquired from multiple different viewpoints.

16. The system according to claim 14, characterized in that, The step of extracting features from multiple single-view images to obtain multiple single-view image features includes: Each single-view image is divided into multiple first gridded images. The feature vector of each first gridded image is calculated to obtain gridded image features. The multiple gridded image features of each single-view image are combined into single-view image features. The grid division method of the first gridded images of the multiple single-view images is consistent. Feature vectors are extracted from each of the single-view images to obtain multiple single-view image features.

17. The system according to claim 16, characterized in that, The overlapping regions between the multiple single-view images are obtained through calculation, including: The overlapping region between the multiple single-view images is calculated by inputting parameters from multiple viewpoint camera devices, wherein the camera device parameters include extrinsic parameters and intrinsic parameters of the camera devices.

18. The system according to claim 17, characterized in that, The calculation of the area ratio of the overlapping region between single-view images is obtained through calculation, including: Based on the overlapping area between the single-view images, a second meshed image corresponding to the overlapping area is determined, and the area ratio occupied by the second meshed image between the single-view images is calculated respectively. Each second meshed image corresponds to the first meshed image of multiple overlapping single-view images.

19. The system according to claim 18, characterized in that, The image surround view feature is obtained by fusing multiple single-view image features based on the overlapping regions between the multiple single-view images and the area ratio of the overlapping regions between the single-view images, including: Based on the area ratio between single-view images in the second gridded image, the feature vectors of the second gridded images that have overlapping relationships between the single-view images are stitched together to obtain the first stitching feature. The first stitching feature is compressed using a vector compression method to obtain overlapping image features, so that the vector dimension of the overlapping image features is consistent with the vector dimension of the second meshed image of the corresponding overlapping region. Based on the spatial relationship between the single-view images, the single-view image features outside the overlapping region and the overlapping image features within the overlapping region are stitched together to obtain the image panoramic features.

20. The system according to claim 19, characterized in that, The vector compression method includes using a multilayer perceptron to compress the vector of the first spliced ​​feature.

21. The system according to any one of claims 14 to 20, characterized in that, The query image features include multiple single-scale image features obtained by generating multiple third-grid images from the query image and extracting features from the multiple third-grid images respectively.

22. The system according to claim 21, characterized in that, The query feature extraction module is used to extract query features of the object to be queried from the query information, including: Input the query information for the object to be queried, which includes the text and image to be queried; The semantic features of the text are extracted based on the text to be queried, and the multiple single-scale image features are extracted based on the image to be queried, wherein the semantic features of the text include one or more keywords describing the object to be queried; Based on the query information, the text semantic features are fused with the multiple single-scale image features to obtain multi-modal fusion features at multiple scales; By fusing the multimodal fusion features at multiple scales, the query features of the object to be queried are obtained.

23. The system according to claim 22, characterized in that, The step of extracting the multiple single-scale image features based on the image to be queried includes: The image to be queried is copied into multiple sub-images to be queried, and each sub-image to be queried is divided into multiple third gridded images according to different scales. The image range contained in the multiple sub-images to be queried is consistent, and the grid division method between the third gridded images at different scales is inconsistent. Multiple single-scale image features are generated by feature extraction from multiple third-mesh images, wherein the multiple single-scale image features of the query image are independent of each other.

24. The system according to claim 23, characterized in that, The step of fusing the text semantic features with corresponding single-scale image features to obtain multi-scale multimodal fusion features includes: The text semantic features and the single-scale image features are used to calculate feature similarity to obtain a weighting factor; The weighting factors and the corresponding single-scale image features are multiplied by a vector to obtain multimodal fusion features; By performing feature similarity calculation and vector multiplication on the single-scale image features at different scales and the text semantic features, the multimodal fusion features at multiple scales are obtained respectively.

25. The system according to claim 24, characterized in that, The process of fusing the multimodal fusion features at multiple scales to obtain the query features of the object to be queried includes: GEM transformation is performed on each of the multiple multimodal fusion features, and the second concatenated feature is obtained by feature concatenation. The vector dimension of the second concatenated feature is reduced to the same as the vector dimension of the data pool feature by the feature dimensionality reduction method, thereby obtaining the query feature of the object to be queried. The feature dimensionality reduction method includes principal component analysis.

26. The system according to claim 25, characterized in that, Before performing the feature dimensionality reduction, the second spliced ​​features with similar characteristics are clustered together using cluster analysis.

27. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores instructions, and the processor, when executing the instructions in the memory, is capable of implementing the method as claimed in any one of claims 1 to 13.

28. A non-volatile storage medium, characterized in that, It includes multiple instructions, which, when executed by an electronic device, enable the implementation of the method as claimed in any one of claims 1 to 13.

Images