Method, device, equipment and medium for spatial positioning based on video images

Abstract

The spatial positioning method, device, equipment and medium based on video images provided in this application relate to the field of artificial intelligence technology. In this application, first, a target video formed by image acquisition for a target object is obtained, and target device data corresponding to the target video is obtained; secondly, feature encoding is performed on the target video to form a video dimension coding feature corresponding to the target video; then, feature encoding is performed on the target device data to form a device dimension coding feature corresponding to the target device data; finally, feature regression is performed based on the video dimension coding feature and the device dimension coding feature, and the spatial positioning data of the target object is output to achieve spatial positioning of the target object. Based on the above content, the problem of relatively low reliability of spatial positioning in the prior art can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus, device, and medium for spatial positioning based on video images. Background Art

[0002] With the development of artificial intelligence technology, video image processing, target detection, and spatial positioning technologies have gradually been widely used. In many application scenarios, accurate spatial positioning data is crucial for realizing technologies such as precise navigation, intelligent monitoring, and augmented reality (AR). Existing technologies are usually based on traditional visual sensors or positioning devices such as GPS and IMU to achieve spatial positioning, but these methods often face deficiencies in reliability and accuracy. Especially in complex environments, positioning errors are large, resulting in inaccurate and unstable positioning results. Most existing spatial positioning methods based on video images rely on extracting features from video images and estimating spatial positions by comparing them with preset maps or environmental models. A major problem with these methods is that the positioning accuracy of target objects is low, especially in harsh conditions such as multi-target environments, occlusion, or low light. The extraction and matching of image features will be greatly affected, resulting in instability of spatial positioning data. Therefore, there is a problem of relatively low reliability of spatial positioning. Summary of the Invention

[0003] In view of this, the purpose of the present application is to provide a spatial positioning method and apparatus, device and medium based on video images, so as to improve the problem of relatively low reliability of spatial positioning in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solutions:

[0005] A spatial positioning method based on video images, comprising:

[0006] Obtaining a target video formed by performing image acquisition on a target object, and obtaining target device data corresponding to the target video, wherein the target video includes multiple video image frames, each of which has the target object, and the target device data refers to device parameters of the image acquisition device used to acquire the target video during image acquisition;

[0007] Performing feature encoding on the target video to form a video dimension encoding feature corresponding to the target video, wherein the video dimension encoding feature has semantic information in the target video;

[0008] Performing feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data, wherein the device dimension encoding feature has semantic information in the target device data, and the target video and the target device data have different feature encodings;

[0009] Feature regression is performed based on the video dimension coding features and the device dimension coding features, and spatial positioning data of the target object is output to achieve spatial positioning of the target object.

[0010] In a preferred embodiment of the present application, in the above-mentioned spatial positioning method based on video images, the step of encoding the target video to form a video dimension encoding feature corresponding to the target video includes:

[0011] performing differential processing on the target video to form a differential video corresponding to the target video, wherein for each differential image frame in the differential video, the differential image frame is obtained based on a difference between a video image frame of a corresponding time sequence in the target video and a previous video image frame, and pixel values of a video image frame previous to the first video image frame are all 0;

[0012] Using a first encoding subunit included in a video encoding unit in a target space positioning model, feature encoding is performed on the target video to form a first video dimension local feature corresponding to the target video, wherein the video encoding unit further includes a second encoding subunit and an encoding aggregation subunit;

[0013] Using the second encoding subunit, feature encoding is performed on the differential video to form a second video dimension local feature corresponding to the differential video;

[0014] The encoding aggregation subunit is used to perform encoding aggregation on the first video dimension local features and the second video dimension local features to form video dimension encoding features corresponding to the target video.

[0015] In a preferred embodiment of the present application, in the above-mentioned spatial positioning method based on video images, the step of performing feature encoding on the target video using the first encoding subunit included in the video encoding unit in the target spatial positioning model to form a first video dimension local feature corresponding to the target video includes:

[0016] The target video is loaded so that a first encoding subunit included in a video encoding unit in a target space positioning model acquires the target video;

[0017] Convolving each video image frame in the target video, outputting a video image convolution feature corresponding to each video image frame, and clustering the output multiple video image convolution features to output multiple center image convolution features, wherein the number of the multiple center image convolution features is less than the number of the multiple video image convolution features;

[0018] Performing serialization processing on the multiple central image convolution features to form a video feature sequence corresponding to the multiple central image convolution features;

[0019] Based on the sequential relationship represented by the video feature sequence, deep attention processing is performed on the multiple central image convolution features in sequence to form a first video dimension local feature corresponding to the target video, wherein, in the deep attention processing, the number of times each central image convolution feature is subjected to attention processing is equal to the serial number of the central image convolution feature in the video feature sequence, and the attention processing corresponding to the first central image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent central image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1.

[0020] In a preferred embodiment of the present application, in the above-mentioned spatial positioning method based on video images, the step of using the second encoding subunit to perform feature encoding on the differential video to form a second video dimension local feature corresponding to the differential video includes:

[0021] Loading the differential video so that the second encoding subunit obtains the differential video;

[0022] Convolving each differential image frame in the differential video, outputting a differential image convolution feature corresponding to each differential image frame, and serializing the outputted multiple differential image convolution features to form a differential feature sequence corresponding to the multiple differential image convolution features;

[0023] Based on the sequential relationship represented by the differential feature sequence, deep attention processing is performed on the multiple differential image convolution features in sequence to form local features of the second video dimension corresponding to the differential video, wherein, in the deep attention processing, the number of times each differential image convolution feature is subjected to attention processing is equal to the serial number of the differential image convolution feature in the differential feature sequence, and the attention processing corresponding to the first differential image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent differential image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1.

[0024] In a preferred embodiment of the present application, in the above-mentioned video image-based spatial positioning method, the step of using the encoding aggregation subunit to encode and aggregate the first video dimension local features and the second video dimension local features to form the video dimension encoding features corresponding to the target video includes:

[0025] Loading the first video dimension local features and the second video dimension local features, so that the encoding aggregation subunit obtains the first video dimension local features and the second video dimension local features;

[0026] Performing a plurality of different downsampling processes on the first video dimension local features and the second video dimension local features respectively to achieve feature compression, thereby forming a corresponding plurality of first downsampled features and a corresponding plurality of second downsampled features;

[0027] For each of the first down-sampled features, the first down-sampled feature and the corresponding second down-sampled feature are concatenated to form a corresponding down-sampled concatenated feature, and the down-sampled concatenated feature is convolved and pooled to form a corresponding feature to be aggregated, wherein the feature to be aggregated corresponding to each of the first down-sampled features has the same feature size:

[0028] An addition operation is performed on the features to be aggregated corresponding to each of the first down-sampled features to form a video dimension coding feature corresponding to the target video.

[0029] In a preferred embodiment of the present application, in the above-mentioned spatial positioning method based on video images, the step of performing feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data includes:

[0030] Loading the target device data so that the device coding unit in the target space positioning model obtains the target device data;

[0031] The target device data is word-embedded to form a device dimension embedding feature corresponding to the target device data, and the distribution position embedding feature corresponding to each segmented word in the target device data is integrated into the device dimension embedding feature to form a device dimension coding feature corresponding to the target device data.

[0032] In a preferred embodiment of the present application, in the above-mentioned spatial positioning method based on video images, the step of performing feature regression based on the video dimension coding features and the device dimension coding features, and outputting the spatial positioning data of the target object to achieve spatial positioning of the target object includes:

[0033] Loading the video dimension coding feature and the device dimension coding feature so that a feature regression unit in a target space positioning model obtains the video dimension coding feature and the device dimension coding feature;

[0034] performing feature space conversion processing on the device dimension coding feature to form a device dimension conversion feature corresponding to the device dimension coding feature, and performing cross attention processing on the video dimension coding feature based on the device dimension conversion feature to form a target multi-dimensional feature;

[0035] Fully connected the target multi-dimensional features and output the spatial positioning data of the target object to achieve the spatial positioning of the target object.

[0036] The present application also provides a spatial positioning device based on video images, comprising:

[0037] a data acquisition module, configured to acquire a target video formed by performing image acquisition on a target object, and acquire target device data corresponding to the target video, wherein the target video includes a plurality of video image frames, each of which contains the target object, and the target device data refers to device parameters of the image acquisition device used to acquire the target video during image acquisition;

[0038] A video encoding module, configured to perform feature encoding on the target video to form a video dimension encoding feature corresponding to the target video, wherein the video dimension encoding feature has semantic information in the target video;

[0039] a data encoding module, configured to perform feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data, wherein the device dimension encoding feature has semantic information in the target device data, and the target video and the target device data have different feature encodings;

[0040] A feature regression module is used to perform feature regression based on the video dimension coding feature and the device dimension coding feature, and output the spatial positioning data of the target object to achieve spatial positioning of the target object.

[0041] Based on the above, the present application further provides an electronic device, including:

[0042] memory for storing computer programs;

[0043] The processor connected to the memory is used to execute the computer program stored in the memory to implement the above-mentioned spatial positioning method based on video images.

[0044] On the basis of the above, the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is run, each step of the above-mentioned video image-based spatial positioning method is executed.

[0045] The spatial positioning method, apparatus, device and medium based on video images provided in the present application first obtain a target video formed by performing image acquisition on a target object, and obtain target device data corresponding to the target video; secondly, feature encoding is performed on the target video to form a video dimension encoding feature corresponding to the target video; then, feature encoding is performed on the target device data to form a device dimension encoding feature corresponding to the target device data; finally, feature regression is performed based on the video dimension encoding feature and the device dimension encoding feature, and the spatial positioning data of the target object is output to achieve spatial positioning of the target object. Based on the above content, since the basis for feature regression includes not only the potential semantic information in the video image, but also the potential semantic information in the device parameters, the reliability of the feature regression can be higher, so that the reliability of the obtained spatial positioning data is also higher, thereby improving the problem of relatively low reliability of spatial positioning in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings.

[0047] Figure 1 This is a structural block diagram of the electronic device provided in an embodiment of the present application.

[0048] Figure 2 A flowchart of a video image-based spatial positioning method provided in an embodiment of the present application.

[0049] Figure 3 A schematic diagram of deep attention processing provided in an embodiment of the present application.

[0050] Figure 4 A block diagram of a video image-based spatial positioning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0053] like Figure 1 As shown, an embodiment of the present application provides an electronic device, wherein the electronic device may include a memory, a processor, and a spatial positioning device based on video images.

[0054] In detail, the memory and the processor are electrically connected directly or indirectly to realize data transmission or interaction. For example, the memory and the processor can be electrically connected through one or more communication buses or signal lines. The spatial positioning device based on video images includes at least one software function module stored in the memory in the form of software or firmware. The processor is used to execute the executable computer program stored in the memory, for example, the software function module and computer program included in the spatial positioning device based on video images, so as to realize the spatial positioning method based on video images provided in the embodiment of the present application.

[0055] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0056] Furthermore, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0057] I understand. Figure 1The structure shown is only for illustration, and the electronic device may also include Figure 1 More or fewer components than shown, or with Figure 1 The different configurations shown, for example, may also include a communication unit for exchanging information with other devices (such as image acquisition devices, etc.).

[0058] Combine Figure 2 The embodiment of the present application further provides a video image-based spatial positioning method applicable to the above-mentioned electronic device. The method steps defined in the process related to the video image-based spatial positioning method can be implemented by the electronic device.

[0059] The following will Figure 2 The specific process shown is explained in detail.

[0060] Step S110 , obtaining a target video formed by performing image acquisition on a target object, and obtaining target device data corresponding to the target video.

[0061] In an embodiment of the present application, the electronic device can obtain a target video formed by performing image acquisition on a target object, and obtain target device data corresponding to the target video. The target video includes multiple video image frames, each of which contains the target object (such as a person, device, animal, etc.), and the target device data refers to the device parameters of the image acquisition device that acquires the target video during image acquisition, such as focal length, shooting angle, distortion coefficients, etc.

[0062] Step S120 , performing feature encoding on the target video to form a video dimension encoding feature corresponding to the target video.

[0063] In an embodiment of the present application, after acquiring the target video, the electronic device may perform feature encoding on the target video to form a video dimension encoding feature corresponding to the target video. The video dimension encoding feature includes semantic information from the target video, that is, the semantic information from the target video is extracted and represented in the form of a vector.

[0064] Step S130 , performing feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data.

[0065] In an embodiment of the present application, after acquiring the target device data, the electronic device may perform feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data. The device dimension encoding feature includes semantic information in the target device data, and the target video and the target device data have different feature encodings. That is, the semantic information in the target device data is extracted and represented in the form of a vector.

[0066] Step S140 , performing feature regression based on the video dimension coding feature and the device dimension coding feature, and outputting the spatial positioning data of the target object to achieve spatial positioning of the target object.

[0067] In an embodiment of the present application, after obtaining the video dimension encoding feature and the device dimension encoding feature, the electronic device can perform feature regression based on the video dimension encoding feature and the device dimension encoding feature, and output the spatial positioning data of the target object to achieve spatial positioning of the target object. It is understood that in an alternative embodiment, the spatial positioning data can be three-dimensional data, namely longitude, latitude, and altitude.

[0068] Based on the above content, since the basis for feature regression includes not only the latent semantic information in the video image, but also the latent semantic information in the device parameters, the reliability of feature regression can be higher, so that the reliability of the obtained spatial positioning data is also higher, thereby improving the problem of relatively low reliability of spatial positioning in the existing technology.

[0069] It should be noted that for step S120 , the specific method of feature encoding the target video is not limited and can be selected accordingly according to actual needs.

[0070] For example, in an alternative embodiment, in order to be able to extract more detailed information from the target video and make the subsequent regression analysis more reliable, the above-mentioned step S120 can further include step S121, step S122, step S123 and step S124, and the specific content of each step is described below.

[0071] Step S121 : performing differential processing on the target video to form a differential video corresponding to the target video.

[0072] In an embodiment of the present application, the target video can be subjected to differential processing to form a differential video corresponding to the target video. For each differential image frame in the differential video, the differential image frame is obtained based on the difference between the corresponding time sequence of the video image frame in the target video and the previous video image frame, and the pixel values of the previous video image frame of the first video image frame are all 0. In other words, the first video image frame can be used as the first differential image frame, and the second video image frame can be subtracted from the first video image frame to obtain the second differential image frame. Similarly, the last video image frame can be subtracted from the second-to-last video image frame to obtain the last differential image frame.

[0073] Step S122: Using the first encoding subunit included in the video encoding unit in the target space positioning model, feature encoding is performed on the target video to form a first video dimension local feature corresponding to the target video.

[0074] In an embodiment of the present application, the first encoding subunit included in the video encoding unit in the target space positioning model can be used to perform feature encoding on the target video to form a first video dimension local feature corresponding to the target video. The video encoding unit also includes a second encoding subunit and an encoding aggregation subunit. The target space positioning model is a trained neural network. During the training process, the parameters of the neural network can be trained based on sample videos, sample device data, and spatial positioning labels. The specific training process will not be described here one by one. Please refer to the relevant existing technology and the relevant explanations of feature encoding and feature regression in the following text.

[0075] Step S123: Utilize the second encoding subunit to perform feature encoding on the differential video to form a second video dimension local feature corresponding to the differential video.

[0076] In the embodiment of the present application, the second encoding subunit may also be used to perform feature encoding on the differential video to form a second video dimension local feature corresponding to the differential video.

[0077] Step S124: Utilize the encoding aggregation subunit to perform encoding aggregation on the first video dimension local features and the second video dimension local features to form video dimension encoding features corresponding to the target video.

[0078] In an embodiment of the present application, after obtaining the first video dimension local features and the second video dimension local features, the encoding aggregation subunit can be used to encode and aggregate the first video dimension local features and the second video dimension local features to form a video dimension encoding feature corresponding to the target video. In this way, the video dimension encoding feature can carry not only the global semantic information of the target video, but also the semantic information of the difference between image frames (i.e., the change of the target object), thereby capturing more detailed information.

[0079] It is understandable that, in the above step S122, the specific manner of forming the local features of the first video dimension corresponding to the target video is not limited. For example, in an alternative embodiment, in order to capture more important information and abstract features (i.e., potential semantic information) in the target video, the above step S122 may further include the following:

[0080] First, the target video may be loaded so that the first encoding subunit included in the video encoding unit in the target space positioning model acquires the target video;

[0081] Secondly, convolution can be performed on each video image frame in the target video, and the video image convolution features corresponding to each video image frame can be output. The output multiple video image convolution features are clustered to output multiple center image convolution features, wherein the number of the multiple center image convolution features is less than the number of the multiple video image convolution features. In this way, the amount of data for subsequent processing can be reduced to a certain extent. In addition, the first encoding subunit can include a convolutional network layer to convolve the video image frame, and the clustering algorithm is not restricted. For example, a number of centers can be specified, and then a corresponding number of cluster centers can be randomly determined. Then, based on the principle of minimum feature distance, each video image convolution feature can be assigned to the set corresponding to the corresponding cluster center. Then, the average value of the feature distance between each video image convolution feature and the cluster center is calculated respectively. In this way, multiple new cluster centers are determined through continuous iteration, thereby obtaining multiple average values. Finally, the multiple cluster centers with the smallest average value are determined as the output multiple center image convolution features.

[0082] Then, the multiple central image convolution features may be serialized (for example, they may be sorted according to a sequence relationship between corresponding video image frames) to form a video feature sequence corresponding to the multiple central image convolution features;

[0083] Finally, based on the order relationship represented by the video feature sequence, the multiple central image convolution features can be sequentially subjected to deep attention processing to form the first video dimension local features corresponding to the target video, wherein, in the deep attention processing, the number of times each central image convolution feature is subjected to attention processing is equal to the serial number of the central image convolution feature in the video feature sequence, and the attention processing corresponding to the first central image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent central image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1. Combined Figure 3 As shown, for the first central image convolution feature, the first central image convolution feature can be self-attention processed to obtain the first output feature; for the second central image convolution feature, the second central image convolution feature can be self-attention processed once to obtain the corresponding self-attention feature, and then, based on the first output feature, the self-attention feature can be cross-attention processed to obtain the second output feature; for the third central image convolution feature, the third central image convolution feature can be cascaded twice with self-attention processed (i.e., the third central image convolution feature is self-attention processed once to obtain an attention feature, and then the attention feature is self-attention processed again) to obtain the corresponding self-attention feature, and then, based on the second output feature, the self-attention feature is cross-attention processed to obtain the third output feature, and so on, to obtain the last output feature. Based on this, it can be ensured that the depth between the two features of each cross-attention processing is consistent, thereby improving the reliability of the cross-attention processing. In addition, after obtaining the last output feature, since the last output feature actually carries the semantic information in the convolution features of each central image, the last output feature can be directly used as the local feature of the first video dimension. Alternatively, in other embodiments, in order to avoid the problem of semantic distortion caused by excessive depth of deep attention processing, the last output feature and the multiple center image convolution features can be added to obtain the first video dimension local feature.

[0084] It is understandable that, in the above step S123, the specific manner of forming the local features of the second video dimension corresponding to the differential video is not limited. For example, in an alternative embodiment, in order to capture more important information and abstract features (i.e., potential semantic information) in the target video, the above step S123 may further include the following:

[0085] First, the differential video may be loaded so that the second encoding subunit obtains the differential video;

[0086] Secondly, convolution can be performed on each differential image frame in the differential video, and differential image convolution features corresponding to each differential image frame can be output. The output multiple differential image convolution features are serialized (they can be sorted according to the order of the corresponding differential image frames) to form a differential feature sequence corresponding to the multiple differential image convolution features. Exemplarily, the second encoding subunit can have a convolutional network layer for performing convolution on the differential image frames.

[0087] Then, based on the order relationship represented by the differential feature sequence, deep attention processing can be performed on the multiple differential image convolution features in sequence to form local features of the second video dimension corresponding to the differential video, wherein, in the deep attention processing, the number of times each differential image convolution feature is subjected to attention processing is equal to the serial number of the differential image convolution feature in the differential feature sequence, and the attention processing corresponding to the first differential image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent differential image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1. The specific method of deep attention processing can refer to the relevant description above.

[0088] It is understandable that, in the above step S124, the specific manner of encoding and aggregating the first video dimension local features and the second video dimension local features is not limited. For example, in an alternative embodiment, in order to enable the formed video dimension encoding features to fully represent the semantic information in the first video dimension local features and the second video dimension local features, the above step S124 may further include the following contents:

[0089] First, the first video dimension local feature and the second video dimension local feature may be loaded, so that the encoding aggregation subunit obtains the first video dimension local feature and the second video dimension local feature;

[0090] Secondly, a plurality of different downsampling processes (corresponding downsampling parameters may be different) may be performed on the first video dimension local features and the second video dimension local features respectively to achieve feature compression, thereby forming a plurality of corresponding first downsampling features and a plurality of second downsampling features;

[0091] Then, for each of the first down-sampled features, the first down-sampled feature and the corresponding second down-sampled feature (i.e., the second down-sampled feature obtained based on the same down-sampling process) can be spliced to form a corresponding down-sampled spliced feature, and the down-sampled spliced feature can be convolved and pooled to form a corresponding feature to be aggregated, wherein the feature size of the feature to be aggregated corresponding to each of the first down-sampled features is the same (by configuring parameters such as the step size of the convolution and pooling, the feature size of the output features to be aggregated is the same):

[0092] Finally, the features to be aggregated corresponding to each of the first down-sampling features can be added together to form the video dimension coding features corresponding to the target video; for example, if the feature sizes of the features to be aggregated finally output are different, corresponding adjustments can be made so that the feature sizes are the same, and then the addition operation can be performed to obtain the corresponding video dimension coding features.

[0093] It should be noted that for step S130 , the specific method of performing feature encoding on the target device data is not limited and can be selected accordingly according to actual needs.

[0094] For example, in an alternative embodiment, in order to improve the semantic representation accuracy of the device dimension coding feature obtained by feature coding, the above-mentioned step S130 may include:

[0095] First, the target device data may be loaded so that the device coding unit in the target space positioning model acquires the target device data;

[0096] Secondly, the target device data can be subjected to word embedding processing to form a device dimension embedding feature corresponding to the target device data, and the distribution position embedding feature corresponding to each segmented word in the target device data (which can also be obtained through word embedding processing) can be integrated into the device dimension embedding feature to form a device dimension coding feature corresponding to the target device data; exemplarily, the device coding unit can include a word embedding model, so that the target device data can be subjected to word segmentation processing to obtain a plurality of corresponding words, and then, for each word and each word's position in the target device data, word embedding processing can be performed to obtain corresponding word embedding features and position embedding features, and then, the word embedding features and position embedding features can be added to obtain corresponding word coding features, and finally, the word coding features corresponding to each word can be spliced to obtain the corresponding device dimension coding features. Alternatively, in other embodiments, after splicing the word coding features corresponding to each word, the obtained splicing features can be subjected to self-attention processing to obtain the corresponding device dimension coding features.

[0097] It should be noted that for step S140 , the specific method of performing feature regression based on the video dimension coding features and the device dimension coding features is not limited and can be selected accordingly according to actual needs.

[0098] For example, in an alternative embodiment, in order to fully utilize the semantic information in the video dimension encoding feature and the device dimension encoding feature during the feature regression process to make the feature regression more reliable, the above-mentioned step S140 may include the following contents:

[0099] First, the video dimension coding feature and the device dimension coding feature may be loaded so that a feature regression unit in a target space positioning model acquires the video dimension coding feature and the device dimension coding feature;

[0100] Secondly, the device dimension coding feature can be subjected to feature space conversion processing to form a device dimension conversion feature corresponding to the device dimension coding feature, and, based on the device dimension conversion feature, the video dimension coding feature can be subjected to cross attention processing to form a target multi-dimensional feature; illustratively, the feature regression unit can include linear mapping parameters, such as a weight parameter and a bias parameter, so that after multiplying the device dimension coding feature by the weight parameter, the multiplication result can be added to the bias parameter to obtain the corresponding device dimension conversion feature;

[0101] Then, the target multidimensional features can be fully connected to output the spatial positioning data of the target object to achieve the spatial positioning of the target object. For example, when the output spatial positioning data belongs to three-dimensional coordinates, the fully connected processing can map the target multidimensional features into a vector, which includes three parameters, such as (a, b, c), where a can be used to characterize accuracy, b can be used to characterize dimension, and c can be used to characterize height.

[0102] Combine Figure 4 The embodiment of the present application further provides a video image-based spatial positioning device applicable to the above-mentioned electronic device. The video image-based spatial positioning device may include a data acquisition module, a video encoding module, a data encoding module, and a feature regression module.

[0103] In detail, the data acquisition module can be used to acquire a target video formed by performing image acquisition on a target object, and acquire target device data corresponding to the target video, wherein the target video includes multiple video image frames, each of which has the target object, and the target device data refers to the device parameters of the image acquisition device that acquires the target video when performing image acquisition. In the embodiment of the present application, the data acquisition module can be used to perform Figure 2 As shown in step S110, for the relevant content of the data acquisition module, reference may be made to the above description of step S110.

[0104] In detail, the video encoding module can be used to perform feature encoding on the target video to form a video dimension encoding feature corresponding to the target video, wherein the video dimension encoding feature has semantic information in the target video. Figure 2 As shown in step S120, for the relevant content of the video encoding module, reference can be made to the above description of step S120.

[0105] In detail, the data encoding module can be used to perform feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data, wherein the device dimension encoding feature has semantic information in the target device data, and the target video and the target device data have different feature encodings. In the embodiment of the present application, the data encoding module can be used to perform Figure 2 As shown in step S130, for the relevant content of the data encoding module, reference can be made to the above description of step S130.

[0106] In detail, the feature regression module can be used to perform feature regression based on the video dimension coding feature and the device dimension coding feature, and output the spatial positioning data of the target object to achieve the spatial positioning of the target object. Figure 2 As shown in step S140, for the relevant content of the feature regression module, reference can be made to the above description of step S140.

[0107] In an embodiment of the present application, corresponding to the above-mentioned spatial positioning method based on video images applied to the electronic device, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is run, the various steps of the spatial positioning method based on video images are executed.

[0108] The steps executed when the aforementioned computer program is running will not be described in detail here, and reference may be made to the above explanation of the spatial positioning method based on video images.

[0109] In summary, the spatial positioning method and apparatus, device and medium based on video images provided by the present application first obtain a target video formed by image acquisition for a target object, and obtain target device data corresponding to the target video; secondly, feature encode the target video to form a video dimension encoding feature corresponding to the target video; then, feature encode the target device data to form a device dimension encoding feature corresponding to the target device data; finally, feature regression is performed based on the video dimension encoding feature and the device dimension encoding feature, and the spatial positioning data of the target object is output to achieve spatial positioning of the target object. Based on the above content, since the basis for feature regression includes not only the latent semantic information in the video image, but also the latent semantic information in the device parameters, the reliability of the feature regression can be higher, so that the reliability of the obtained spatial positioning data is also higher, thereby improving the problem of relatively low reliability of spatial positioning in the prior art.

[0110] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0112] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or partly contributed to the prior art or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0113] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A spatial positioning method based on video images, characterized in that: include: Obtaining a target video formed by performing image acquisition on a target object, and obtaining target device data corresponding to the target video, wherein the target video includes multiple video image frames, each of which has the target object, and the target device data refers to device parameters of the image acquisition device used to acquire the target video during image acquisition; performing differential processing on the target video to form a differential video corresponding to the target video, wherein for each differential image frame in the differential video, the differential image frame is obtained based on a difference between a video image frame of a corresponding time sequence in the target video and a previous video image frame, and pixel values of a video image frame previous to the first video image frame are all 0; Using a first encoding subunit included in a video encoding unit in a target space positioning model, feature encoding is performed on the target video to form a first video dimension local feature corresponding to the target video, wherein the video encoding unit further includes a second encoding subunit and an encoding aggregation subunit; The differential video is loaded so that the second encoding subunit obtains the differential video; convolution is performed on each differential image frame in the differential video, and the differential image convolution features corresponding to each differential image frame are output, and the output multiple differential image convolution features are serialized to form a differential feature sequence corresponding to the multiple differential image convolution features; based on the order relationship represented by the differential feature sequence, deep attention processing is performed on the multiple differential image convolution features in sequence to form a second video dimension local feature corresponding to the differential video, wherein, in the deep attention processing, the number of times each differential image convolution feature is subjected to attention processing is equal to the serial number of the differential image convolution feature in the differential feature sequence, and the attention processing corresponding to the first differential image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent differential image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1; Utilizing the encoding aggregation subunit, encoding and aggregating the first video dimension local features and the second video dimension local features to form a video dimension encoding feature corresponding to the target video; Performing feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data, wherein the device dimension encoding feature has semantic information in the target device data, and the target video and the target device data have different feature encodings; Feature regression is performed based on the video dimension coding features and the device dimension coding features, and spatial positioning data of the target object is output to achieve spatial positioning of the target object.

2. The video image-based spatial positioning method according to claim 1, characterized in that: The step of utilizing the first encoding subunit included in the video encoding unit in the target space positioning model to perform feature encoding on the target video to form a first video dimension local feature corresponding to the target video includes: The target video is loaded so that a first encoding subunit included in a video encoding unit in a target space positioning model acquires the target video; Convolving each video image frame in the target video, outputting a video image convolution feature corresponding to each video image frame, and clustering the output multiple video image convolution features to output multiple center image convolution features, wherein the number of the multiple center image convolution features is less than the number of the multiple video image convolution features; Performing serialization processing on the multiple central image convolution features to form a video feature sequence corresponding to the multiple central image convolution features; Based on the sequential relationship represented by the video feature sequence, deep attention processing is performed on the multiple central image convolution features in sequence to form a first video dimension local feature corresponding to the target video, wherein, in the deep attention processing, the number of times each central image convolution feature is subjected to attention processing is equal to the serial number of the central image convolution feature in the video feature sequence, and the attention processing corresponding to the first central image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent central image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1.

3. The video image-based spatial positioning method according to claim 1, characterized in that: The step of using the encoding aggregation subunit to encode and aggregate the first video dimension local features and the second video dimension local features to form the video dimension encoding features corresponding to the target video includes: Loading the first video dimension local features and the second video dimension local features, so that the encoding aggregation subunit obtains the first video dimension local features and the second video dimension local features; Performing a plurality of different downsampling processes on the first video dimension local features and the second video dimension local features respectively to achieve feature compression, thereby forming a corresponding plurality of first downsampled features and a corresponding plurality of second downsampled features; For each of the first down-sampled features, the first down-sampled feature and the corresponding second down-sampled feature are concatenated to form a corresponding down-sampled concatenated feature, and the down-sampled concatenated feature is convolved and pooled to form a corresponding feature to be aggregated, wherein the feature to be aggregated corresponding to each of the first down-sampled features has the same feature size: An addition operation is performed on the features to be aggregated corresponding to each of the first down-sampled features to form a video dimension coding feature corresponding to the target video.

4. The spatial positioning method based on video images according to claim 1, characterized in that: The step of encoding the target device data to form a device dimension encoding feature corresponding to the target device data includes: Loading the target device data so that the device coding unit in the target space positioning model obtains the target device data; The target device data is word-embedded to form a device dimension embedding feature corresponding to the target device data, and the distribution position embedding feature corresponding to each segmented word in the target device data is integrated into the device dimension embedding feature to form a device dimension coding feature corresponding to the target device data.

5. The video image-based spatial positioning method according to any one of claims 1 to 4, characterized in that: The step of performing feature regression based on the video dimension coding feature and the device dimension coding feature, and outputting the spatial positioning data of the target object to achieve spatial positioning of the target object, includes: Loading the video dimension coding feature and the device dimension coding feature so that a feature regression unit in a target space positioning model obtains the video dimension coding feature and the device dimension coding feature; performing feature space conversion processing on the device dimension coding feature to form a device dimension conversion feature corresponding to the device dimension coding feature, and performing cross attention processing on the video dimension coding feature based on the device dimension conversion feature to form a target multi-dimensional feature; Fully connected the target multi-dimensional features and output the spatial positioning data of the target object to achieve the spatial positioning of the target object.

6. A spatial positioning device based on video images, characterized in that: include: a data acquisition module, configured to acquire a target video formed by performing image acquisition on a target object, and acquire target device data corresponding to the target video, wherein the target video includes a plurality of video image frames, each of which contains the target object, and the target device data refers to device parameters of the image acquisition device used to acquire the target video during image acquisition; A video encoding module is used to perform differential processing on the target video to form a differential video corresponding to the target video, wherein for each differential image frame in the differential video, the differential image frame is obtained based on the difference between the corresponding time sequence video image frame in the target video and the previous frame video image frame, and the pixel values of the previous frame video image frame of the first frame video image frame are all 0; using the first encoding subunit included in the video encoding unit in the target space positioning model, the target video is feature-encoded to form a first video dimension local feature corresponding to the target video, wherein the video encoding unit also includes a second encoding subunit and an encoding aggregation subunit; the differential video is loaded so that the second encoding subunit obtains the differential video; convolution is performed on each differential image frame in the differential video respectively, and the differential image convolution feature corresponding to each differential image frame is output, and the output multiple differential images are processed. The convolution features are serialized to form a differential feature sequence corresponding to the multiple differential image convolution features; based on the order relationship represented by the differential feature sequence, the multiple differential image convolution features are sequentially subjected to deep attention processing to form a second video dimension local feature corresponding to the differential video, wherein, in the deep attention processing, the number of times each differential image convolution feature is subjected to attention processing is equal to the serial number of the differential image convolution feature in the differential feature sequence, and the attention processing corresponding to the first differential image convolution feature includes self-attention processing, and the attention processing corresponding to the second and subsequent differential image convolution features includes self-attention processing and cross-attention processing, and the number of cross-attention processing is equal to 1; using the encoding aggregation subunit, the first video dimension local feature and the second video dimension local feature are encoded and aggregated to form the video dimension encoding feature corresponding to the target video; a data encoding module, configured to perform feature encoding on the target device data to form a device dimension encoding feature corresponding to the target device data, wherein the device dimension encoding feature has semantic information in the target device data, and the target video and the target device data have different feature encodings; A feature regression module is used to perform feature regression based on the video dimension coding feature and the device dimension coding feature, and output the spatial positioning data of the target object to achieve spatial positioning of the target object.

7. An electronic device, characterized in that: include: memory for storing computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the video image-based spatial positioning method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when running, executes the video image-based spatial positioning method according to any one of claims 1 to 5.

Images