Construction method and device of spatial reasoning data set, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a spatial reasoning data set construction method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of target videos, the target videos being videos with object motion; for each target video, determining a three-dimensional space trajectory of each object in the target video; on the basis of the three-dimensional space track, questions and answers corresponding to the questions are generated for the multiple motion attributes, the questions and the answers corresponding to the questions serve as question and answer data pairs, and the motion attributes indicate changing attributes of the object in the three-dimensional space; and generating a spatial reasoning data set based on the target video and the plurality of question and answer data pairs corresponding to the target video. According to the embodiment of the invention, the spatial reasoning data set covering a plurality of dynamic scenes and a plurality of motion attributes can be constructed, so that the visual language model understands and reasones spatial relationship changes among objects in the dynamic scenes.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for constructing a spatial reasoning dataset. Background Technology

[0002] Visual language models refer to artificial intelligence models that integrate computer vision and natural language processing. Their core capability is to achieve semantic understanding, association, and generation between images / videos and text. Currently, most visual language models are limited to understanding visual information in two-dimensional space. Although some visual language models can understand and reason in three-dimensional space, i.e., spatial reasoning, these models are mainly designed for static scenes and cannot understand the spatial relationship changes between objects in dynamic scenes.

[0003] Current approaches utilize spatial reasoning datasets, which allow for the understanding and inference of object motion in 3D space, to train visual language models to comprehend and reason about changes in spatial relationships between objects in dynamic scenes. However, existing spatial reasoning datasets primarily target simple, controlled dynamic scenes, and the questions within these datasets only cover simple issues (e.g., "Did the blue ball in the video move?"). Such datasets fail to enable visual language models to truly understand and infer changes in spatial relationships between objects in dynamic scenes. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail in this disclosure. This overview is not intended to limit the scope of the claims.

[0005] This disclosure provides a method, apparatus, electronic device, and storage medium for constructing a spatial reasoning dataset, which can construct a spatial reasoning dataset covering multiple dynamic scenes and multiple motion attributes, so that a visual language model can understand and reason about changes in spatial relationships between objects in a dynamic scene.

[0006] On one hand, embodiments of this disclosure provide a method for constructing a spatial reasoning dataset, including: Acquire multiple target videos, wherein the target videos are videos in which objects are moving; For each of the target videos, determine the three-dimensional spatial trajectory of each object in the target video; Based on the three-dimensional spatial trajectory, questions and corresponding answers are generated for multiple motion attributes, and the questions and corresponding answers are used as question-answer data pairs. The motion attributes indicate the properties of an object that change in three-dimensional space. A spatial reasoning dataset is generated based on the target video and multiple question-and-answer data pairs corresponding to the target video.

[0007] On the other hand, embodiments of this disclosure also provide an apparatus for constructing a spatial reasoning dataset, including: The acquisition module is used to acquire multiple target videos, wherein the target videos are videos in which objects are moving; The determining module, for each of the target videos, is used to determine the three-dimensional spatial trajectory of each object in the target video; The first generation module is used to generate questions and corresponding answers for multiple motion attributes based on the three-dimensional spatial trajectory, and to use the questions and corresponding answers as question-answer data pairs, wherein the motion attributes indicate the attributes that the object has changing in three-dimensional space; The second generation module is used to generate a spatial reasoning dataset based on the target video and multiple question-and-answer data pairs corresponding to the target video.

[0008] Furthermore, the aforementioned determining module is specifically used for: Extract the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame from the target video. The three-dimensional spatial trajectory of each object in the target video is determined based on the camera pose, the point cloud distribution, the object mask, and the object pose.

[0009] Furthermore, the aforementioned determining module is specifically used for: The category of each object in the target video is determined based on the video description text corresponding to the target video; The target video is uniformly sampled to obtain a video frame sequence, which includes multiple consecutive video frames. Based on the categories of each object in the target video, the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame are extracted from the video frame sequence.

[0010] Furthermore, the aforementioned determining module is specifically used for: The video frame sequence is analyzed using a visual understanding model to obtain the camera pose of the shooting camera and the point cloud distribution of each video frame; The video frame sequence and the category of the object are analyzed using a mask extraction model to obtain the object mask for each video frame; The video frame sequence is analyzed using a pose analysis model to obtain the object pose of each object in each video frame.

[0011] Furthermore, the aforementioned determining module is specifically used for: Based on the object mask in each video frame, the point cloud of the object is filtered out from the point cloud distribution of each video frame. For each object in each video frame, the position of the object in three-dimensional space is determined based on the point cloud of the object; According to the time sequence of the video frames, the positions of each object in each video frame in three-dimensional space are sequentially connected to obtain the three-dimensional spatial trajectory of each object in the target video.

[0012] Furthermore, the aforementioned determining module is specifically used for: Obtain the three-dimensional spatial coordinates of each point in the point cloud of the object; The average value of the three-dimensional spatial coordinates of each point is taken as the position of the object in three-dimensional space.

[0013] Furthermore, the aforementioned first generation module is specifically used for: For each motion attribute, a preset template is populated based on the motion attribute to obtain questions that address the motion attribute. Based on the three-dimensional spatial trajectory, the questions posed regarding the motion attributes are calculated to obtain the answers corresponding to the questions.

[0014] Furthermore, the questions asked regarding the motion attributes include question text and multiple question options; The first generation module mentioned above is specifically used for: The preset template is filled with the motion attributes to obtain the question text that asks questions about the motion attributes. Based on the motion attribute and a preset set of question options, multiple question options corresponding to the motion attribute are determined, wherein the set of question options includes multiple question options corresponding to each motion attribute.

[0015] Furthermore, the preset templates include an absolute perspective template and a relative perspective template. The absolute perspective template is a question template that asks questions about motion attributes based on a static reference frame, while the relative perspective template is a question template that asks questions about motion attributes based on a dynamic reference frame. The first generation module mentioned above is specifically used for: When the preset template is an absolute perspective template, the static reference frame, the target object, and the time range of the question are determined. The absolute perspective template is then filled based on the static reference frame, the target object, the time range, and the motion attribute to obtain the question text that asks about the motion attribute. When the preset template is a relative perspective template, the dynamic reference frame, the target object, and the time range of the question are determined. The relative perspective template is then filled based on the dynamic reference frame, the target object, the time range, and the motion attribute to obtain the question text that asks about the motion attribute.

[0016] Furthermore, the aforementioned first generation module is specifically used for: The three-dimensional spatial trajectories of each object are transformed into the reference frame applicable to the problem, resulting in the transformed three-dimensional spatial trajectories of each object. Identify multiple consecutive video frames that correspond to the time range addressed in the problem; For adjacent video frames, the changes in motion attributes within the adjacent video frames are calculated based on the transformed three-dimensional spatial trajectory to obtain a sub-answer to the problem; By integrating multiple sub-answers to the question, the answer corresponding to the question is obtained.

[0017] Furthermore, the aforementioned first generation module is specifically used for: The three-dimensional spatial trajectory is analyzed using a question generation model to generate questions that address multiple motion attributes and corresponding answers.

[0018] Furthermore, the aforementioned acquisition module is specifically used for: Obtain a video dataset, which includes multiple videos; Target videos containing moving objects are selected from the video dataset.

[0019] Furthermore, the aforementioned acquisition module is specifically used for: For each video in the video dataset, obtain the video description text corresponding to the video, where the video description text is text describing the events that occur in the video; The system detects whether there is a description of object movement in the video description text. If there is a description of object movement in the video description text, the video is identified as the target video.

[0020] Furthermore, the aforementioned acquisition module is specifically used for: For each video in the video dataset, obtain multiple video frames of the video; Based on multiple video frames, it is determined whether there is motion in the video. When there is motion in the video, the video is identified as the target video.

[0021] Furthermore, the aforementioned second generation module is specifically used for: A first number of target videos and the question-and-answer data pairs corresponding to the target videos are used as a training dataset, and a second number of target videos and the question-and-answer data pairs corresponding to the target videos are used as an evaluation dataset, wherein the second number is less than the first number; The training dataset and the evaluation dataset are integrated to obtain the spatial reasoning dataset.

[0022] Furthermore, the apparatus for constructing the spatial reasoning dataset also includes a training module, which is specifically used for: The visual language model is trained based on the aforementioned spatial reasoning dataset; The visual language model reasones about the problem in the following way: Receive spatial reasoning questions and videos; The spatial reasoning problem is transformed into a text feature vector, and the spatial reasoning video is transformed into a three-dimensional geometric feature vector; The text feature vector is semantically compressed to obtain the compressed text feature vector; Extract the target three-dimensional geometric feature vector that is semantically related to the compressed text feature vector from the three-dimensional geometric feature vector; Based on the text feature vector and the target three-dimensional geometric feature vector, a fusion reasoning is performed to obtain the solution corresponding to the spatial reasoning problem.

[0023] Furthermore, the aforementioned training module is specifically used for: The spatial reasoning video is converted into a visual feature vector; Based on the visual feature vector, extract the three-dimensional geometric feature vector from the spatial reasoning video; The process of fusing and reasoning based on the text feature vector and the target 3D geometric feature vector to obtain a solution corresponding to the spatial reasoning problem includes: Based on the text feature vector, the visual feature vector, and the target three-dimensional geometric feature vector, a fusion reasoning is performed to obtain a solution corresponding to the spatial reasoning problem.

[0024] On the other hand, this disclosure also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method for constructing a spatial reasoning dataset.

[0025] On the other hand, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the above-described method for constructing a spatial reasoning dataset.

[0026] On the other hand, this disclosure also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method for constructing the spatial inference dataset described above.

[0027] This disclosure provides at least the following beneficial effects: By acquiring multiple target videos containing moving objects, a massive amount of effective videos covering various dynamic scenes can be obtained, not just simple, controlled scenes. Then, for each target video, the spatial relationship changes between objects are clarified by determining the three-dimensional spatial trajectories of each object in the target video. When the target video covers multiple dynamic scenes, this disclosure also generates questions and corresponding answers based on the three-dimensional spatial trajectories, addressing multiple motion attributes (e.g., distance, direction, speed, pose). These questions and answers are then used as question-and-answer data pairs, ensuring that the data covers not only multiple dynamic scenes but also multiple motion attributes. Therefore, compared to existing spatial reasoning datasets that cover simple, controlled scenes and only some simple questions, this disclosure's spatial reasoning dataset, generated based on target videos and multiple question-and-answer data pairs, covers multiple dynamic scenes and multiple motion attributes. Training a visual language model with this spatial reasoning dataset enables the model to understand and reason about changes in spatial relationships between objects in dynamic scenes.

[0028] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing this disclosure. Attached Figure Description

[0029] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0030] Figure 1 A schematic diagram illustrating an optional implementation environment provided for an embodiment of this disclosure; Figure 2 A schematic diagram illustrating the application of the spatial reasoning dataset construction method provided in the embodiments of this disclosure to the training of a visual understanding model; Figure 3 This is an optional flowchart illustrating a method for constructing a spatial reasoning dataset provided in an embodiment of this disclosure. Figures 4A-4B A schematic diagram of question-and-answer data pairs provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of an optional process for filtering target videos containing moving objects from a video dataset, as provided in an embodiment of this disclosure. Figure 6 This is an optional flowchart illustrating an embodiment of the present disclosure for extracting the camera pose of the camera capturing the target video, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame. Figure 7 This is a schematic diagram of an optional process for generating questions and corresponding answers based on three-dimensional spatial trajectories according to an embodiment of the present disclosure. Figure 8 A schematic diagram comparing the spatial reasoning capabilities of the visual understanding model and other models after the spatial reasoning dataset construction method provided in this embodiment of the disclosure is applied to the training of the visual understanding model; Figure 9 A schematic diagram of an optional structure of the visual understanding model provided in this embodiment of the disclosure; Figure 10 A schematic diagram of an optional architecture for a method of constructing a spatial reasoning dataset provided in an embodiment of this disclosure; Figure 11 A schematic diagram of an optional structure for a spatial reasoning dataset construction apparatus provided in an embodiment of this disclosure; Figure 12 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure; Figure 13 This is a partial structural block diagram of a server provided in an embodiment of this disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0032] It should be noted that in the various specific embodiments of this disclosure, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. The target object can be a user. In addition, when embodiments of this disclosure require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this disclosure be obtained.

[0033] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0034] Visual language models (VLMs) are artificial intelligence models that integrate computer vision and natural language processing. Their core capability is to achieve semantic understanding, association, and generation between images / videos and text. Currently, most VLMs are limited to understanding visual information in two-dimensional space. While some VLMs can perform understanding and reasoning in three-dimensional space (spatial reasoning), these models primarily target static scenes and cannot understand changes in spatial relationships between objects in dynamic scenes. Some current solutions use spatial reasoning datasets that demonstrate the understanding and reasoning of object motion in three-dimensional space to train VLMs to understand and reason about changes in spatial relationships between objects in dynamic scenes. However, existing spatial reasoning datasets mostly cover simple, controlled dynamic scenes, and the questions in these datasets only cover simple questions (e.g., "Did the blue ball in the video move?"). Such spatial reasoning datasets cannot enable VLMs to truly understand and reason about changes in spatial relationships between objects in dynamic scenes.

[0035] Based on this, the present disclosure provides a method, apparatus, electronic device and storage medium for constructing a spatial reasoning dataset, which can construct a spatial reasoning dataset covering multiple dynamic scenes and multiple motion attributes, so that a visual language model can understand and reason about changes in spatial relationships between objects in a dynamic scene.

[0036] Reference Figure 1 , Figure 1 This is a schematic diagram of an optional implementation environment provided by an embodiment of the present disclosure. The implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.

[0037] For example, server 102 can respond to a build request sent by terminal 101 by acquiring multiple videos containing moving objects, i.e., target videos. Then, for each target video, the server determines the three-dimensional spatial trajectory of each object in the target video, and generates questions and corresponding answers based on the three-dimensional spatial trajectories, targeting multiple motion attributes (such as distance, speed, direction, pose, etc.), and sets the questions and corresponding answers as question-answer data pairs. Finally, a spatial inference dataset is generated based on each target video and the multiple question-answer data pairs corresponding to each target video.

[0038] Server 102 acquires multiple target videos containing moving objects, resulting in a massive amount of effective videos covering various dynamic scenes, not just simple, controlled scenes. Then, for each target video, the spatial relationships between objects are clarified by determining their 3D spatial trajectories. In cases where the target videos cover multiple dynamic scenes, this embodiment also generates questions and corresponding answers based on the 3D spatial trajectories, addressing multiple motion attributes (e.g., distance, direction, speed, pose). These questions and answers are then used as question-and-answer data pairs, ensuring coverage of not only multiple dynamic scenes but also multiple motion attributes. Therefore, compared to existing spatial reasoning datasets that cover simple, controlled scenes and only some simple questions, this embodiment generates a spatial reasoning dataset based on target videos and corresponding question-and-answer data pairs, covering both dynamic scenes and multiple motion attributes. Training a visual language model with this spatial reasoning dataset enables the model to understand and infer changes in spatial relationships between objects in dynamic scenes.

[0039] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 102 can also be a node server in a blockchain network.

[0040] Terminal 101 may be a mobile phone, computer, smart voice interaction device, smart home appliance, vehicle terminal, etc., but is not limited to these. Terminal 101 and server 102 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the disclosure does not impose any limitations.

[0041] This disclosure can be applied to the training of visual understanding models. The target video and question-answering data pairs in the spatial reasoning dataset constructed by this disclosure are as follows: Figure 2 As shown. In Figure 2 In the image, the top part shows multiple consecutive video frames of the target video, and the bottom part shows a question-and-answer data pair corresponding to the target video. Through the multiple consecutive video frames, it can be seen that the video content is a boy and two dogs running towards the camera. During this process, the spatial relationship between the boy and the two dogs changes, meaning that the dynamic scene targeted by this target video is a relatively complex scene. Based on this, the question in the question-and-answer data pair generated based on the three-dimensional spatial trajectory of the target video in this embodiment is: "During the period from the 3rd to the 16th second, from the perspective of the boy with an initial bounding box of [221,80,246,262], how does the direction of the dog with an initial bounding box of [371,195,386,243] change relative to the dog with an initial bounding box of [323,187,340,236]?" The corresponding answer is "A. (1) Left; (2) Left rear." It can be understood that the above question is a question asking about the motion attribute of direction. This target video includes multiple question-and-answer data pairs, covering questions and answers related to the motion attribute of distance, speed, and pose. This processing ensures the question-and-answer data pairs cover not only various dynamic scenes but also multiple motion attributes. Training a visual understanding model using this spatial reasoning dataset facilitates the visual language model's understanding and inference of spatial relationships between objects in dynamic scenes.

[0042] Reference Figure 3 , Figure 3 This is an optional flowchart illustrating a method for constructing a spatial reasoning dataset provided in this embodiment of the present disclosure. The method for constructing the spatial reasoning dataset can be executed by a server, or by a terminal, or by a server in conjunction with a terminal. The method for constructing the spatial reasoning dataset includes, but is not limited to, the following steps 301 to 304.

[0043] Step 301: Acquire multiple target videos.

[0044] In this context, video refers to two-dimensional video content that contains three-dimensional geometric information of a scene or object. Videos can provide not only the three-dimensional spatial attributes of objects and scenes (such as position, pose, and distance), but also record temporal changes in the scene. The target video is one in which objects are moving. For example, consider videos A, B, and C. Video A shows a cat sleeping on a sofa, and neither the camera nor the cat's position nor the camera's pose changes. Video B shows a person running towards the camera. Video C is a dashcam recording of a vehicle moving at a constant speed. Based on this, videos B and C can be identified as the target videos.

[0045] It should be noted that the embodiments disclosed herein do not limit the source, format, or encoding format of the video (i.e., two-dimensional video content). For example, the video may be taken by a camera device, recorded on the screen, generated by software, or generated by a model. The video format may be QuickTime Movie (MOV), Audio Video Interleave (AVI), or Matroska Video (MKV), and the video encoding format may be H.264 or H.265. In some cases, the video may also come from the Koala-36M [8] general video dataset, which contains a large number of video clips. Compared with conventional video datasets that do not filter videos, the text descriptions corresponding to the video clips in the Koala-36M [8] dataset are more detailed and can cover details such as actions, backgrounds, and object interactions in the video. The quality of the video clips is also higher (low-resolution / content-irrelevant clips are filtered out). Moreover, the video clips in the Koala-36M [8] dataset cover not only simple, controlled scenes, but also a variety of dynamic scenes. By using the Koala-36M [8] dataset as the source of the videos, the final generated spatial reasoning dataset can cover a variety of dynamic scenes.

[0046] Step 302: For each target video, determine the three-dimensional spatial trajectory of each object in the target video.

[0047] The target video consists of multiple video frames arranged chronologically, each representing a specific moment in the target video. Furthermore, each video frame contains rich motion attribute information, including camera pose, the speed and distance of movement of various objects, pose changes, and position changes. By analyzing the motion attribute information of multiple video frames corresponding to the target video, the three-dimensional spatial trajectories of various objects in the target video can be determined.

[0048] In this context, a three-dimensional spatial trajectory refers to the continuous spatial path formed by the change of the center position of an object in a three-dimensional coordinate system over time as the target video changes. Essentially, a three-dimensional spatial trajectory is a combination of the time dimension and the three-dimensional spatial dimension. Through a three-dimensional spatial trajectory, the position of an object at a certain moment can be determined, as well as the trajectory changes of the object over time. Specifically, a three-dimensional spatial trajectory can be represented as T=[(t1,X1,Y1,Z1),(t2,X2,Y2,Z2), ...,(t n , X n ,Y n Z n )]. Among them, t i Represents the timestamp of the i-th video frame; (X i , Y i Z i Let represent the 3D center position of the object in the i-th video frame. This setting allows for a realistic reconstruction of the object's motion in 3D space (e.g., "moving forward and upward," "rotating and translating along the Z-axis"), rather than a planar representation of a 2D trajectory (projected only onto the X / Y plane). Furthermore, because the 3D trajectory reflects changes in the spatial relationships between objects in the target video, it also allows for accurate prediction of the object's spatial relationships and motion attributes in the short term.

[0049] For example, if the content of target video A is a boy and two dogs running towards the camera, then for target video A, it is necessary to determine the three-dimensional spatial trajectory of the boy, the three-dimensional spatial trajectory of the first dog, and the three-dimensional spatial trajectory of the second dog.

[0050] Step 303: Generate questions and corresponding answers based on the three-dimensional spatial trajectory, and use the questions and corresponding answers as question-answer data pairs.

[0051] Motion attributes refer to attributes used to describe the dynamic changes of an object in three-dimensional space; that is, attributes that indicate changes in the object within three-dimensional space. Motion attributes include distance attributes, direction attributes, pose attributes, velocity attributes, relative relationship attributes, and predictive attributes. Specifically, distance attributes refer to the relative distance between an object and a reference object (e.g., another object, a camera) in three-dimensional space at different points in time, and the temporal trend of distance change (e.g., gradually increasing, gradually decreasing). Direction attributes refer to the changes in the object's three-dimensional relative orientation (e.g., left front, right rear) or the object's own orientation (e.g., the direction a car is facing). Pose attributes refer to the changes in the object's own three-dimensional posture (including position and orientation) (e.g., turning forward, turning left). Velocity attributes refer to the amount of three-dimensional displacement of an object per unit time, and the trend of velocity change (e.g., acceleration, deceleration). Relative relationship attributes refer to the comparison of motion characteristics such as distance, velocity, and direction between multiple objects (e.g., object A's velocity is faster than object B). Predictive attributes refer to the inference of an object's distance, direction, pose, and other characteristics at a future point in time based on its historical motion trajectory. Furthermore, the values ​​of motion attributes will change over time as the target video evolves.

[0052] It is understandable that questions based on 3D spatial trajectory generation that focus on motion attributes refer to questions about the changes in the motion attributes of objects in a target video over a specific time period. For example: Figure 4A As shown, a question based on the motion attributes generated from the 3D spatial trajectory could be, "Between 13 and 35 seconds, how does the distance between the car and the camera, with initial bounding box coordinates [124, 116, 263, 224], change from the camera's perspective?" This indicates that the object being asked in this question is... Figure 4A The car in the box is associated with the distance as its motion attribute, and the time period is from the 13th to the 35th second. For example: Figure 4B As shown, a question based on the motion attributes generated from the 3D spatial trajectory could be: "Between 16 and 18 seconds, from the perspective of the vehicle with initial bounding box coordinates [78, 156, 304, 284], how does the orientation of the vehicle with initial bounding box coordinates [10, 174, 67, 234] change relative to the vehicle with initial bounding box coordinates [78, 156, 304, 284]?" It can be seen that the object targeted by this question is... Figure 4B The cars in the two boxes represent motion attributes of direction and time periods from the 16th to the 18th second.

[0053] Furthermore, the questions generated based on the three-dimensional spatial trajectory for motion attributes include not only the text of the question but also multiple basic answer options corresponding to the text of the question. In some cases, embodiments of this disclosure provide the following basic answer options for the text of the question corresponding to each motion attribute: For the two motion attributes of distance and speed, the following basic answer options can be provided: (1) Remains basically unchanged; (2) Remains basically unchanged and then increases; (3) Remains basically unchanged and then decreases; (4) Gradually increases; (5) Gradually decreases. For the motion attributes of direction, pose, and predicted direction, the following basic answer options can be provided: (1) Forward / backward; (2) Left / right; (3) Up / down. For the motion attribute of comparing speeds (e.g., which is faster, object A or object B), the following basic answer options can be provided: (1) Both are the same; (2) The former is faster; (3) The latter is faster.

[0054] In practice, the basic answer options can be reduced or modified based on the 3D spatial trajectory in the target video, so that the correct answer to the question is included among the multiple basic answer options corresponding to the question's text. For example: it can be based on... Figure 4A The three-dimensional spatial trajectory in the target video is used to modify and delete the basic answer options corresponding to the distance, resulting in answer options including "A. (1) decreases (2) increases", "B. (1) remains unchanged and then increases (2) increases (3) decreases", "C. (1) increases (2) remains unchanged and then decreases (3) decreases" and "D. (1) remains unchanged". This can be based on... Figure 4B The three-dimensional spatial trajectory in the target video is used to modify and delete the basic answer options corresponding to the direction, resulting in answer options including "A. (1) to the right front and up (2) to the left rear", "B. (1) to the left (2) to the left rear", "C. (1) to the right lower and down" and "D. (1) to the left (2) to the left upper".

[0055] It should be noted that because the 3D spatial trajectory clearly defines the motion trajectory of each object in the target video over time, as well as the changes in the spatial relationships between these objects, the answer corresponding to the question asked about the motion attributes can be quickly determined by calculating the 3D spatial trajectory. For example: regarding questions like... Figure 4A The answer to the question can be determined by calculating the three-dimensional spatial trajectory of the vehicle within the box in the target video: "C. (1) Increase (2) Remain unchanged and then decrease (3) Decrease". Regarding the question... Figure 4B The answer to the question can be determined by calculating the three-dimensional spatial trajectory of the vehicles in the two boxes in the target video. The answer is "B. (1) to the left (2) to the left rear".

[0056] It is understood that a question-and-answer data pair is a data structure consisting of the text of a question that asks about motion attributes, the answer options corresponding to the text of the question, and the answer corresponding to the text of the question. When the target video covers multiple dynamic scenes, the question-and-answer data pairs generated based on three-dimensional spatial trajectories, which consist of questions asking about multiple motion attributes and the corresponding answers, not only cover multiple dynamic scenes but also multiple motion attributes.

[0057] Step 304: Generate a spatial reasoning dataset based on the target video and multiple question-and-answer data pairs corresponding to the target video.

[0058] The spatial reasoning dataset refers to data used to train a visual language model, enabling it to possess spatial reasoning capabilities and understand and process changes in the spatial relationships, positions, directions, and distances of objects in dynamic scenes. Specifically, multiple training samples can be generated from the target video and its corresponding question-and-answer pairs (questions based on the 3D spatial trajectories of each object in the target video, posing questions about multiple motion attributes, and the corresponding answers). Each training sample includes the target video and a corresponding question-and-answer pair.

[0059] It is worth noting that, to ensure the correctness and reasonableness of the training samples, they can be further screened and corrected manually. Then, a portion of the training samples is used as the training set for training the spatial reasoning ability of the visual language model, and the remaining samples are used as the evaluation set for assessing the spatial reasoning ability of the visual language model. Furthermore, the number of training samples in the training set is much larger than the number of training samples in the evaluation set. For example, the number of training samples in the training set can be 50,000, while the number of training samples in the evaluation set can be 1,484.

[0060] Based on this, by acquiring multiple target videos containing moving objects, a massive amount of effective videos covering various dynamic scenes can be obtained, not just simple, controlled scenes. Then, for each target video, by determining the three-dimensional spatial trajectories of each object in the target video, the changes in spatial relationships between objects in the target video are clarified. When the target video covers multiple dynamic scenes, this embodiment of the disclosure will also generate questions and corresponding answers based on the three-dimensional spatial trajectories, addressing multiple motion attributes (such as distance, direction, speed, pose, etc.), and use the questions and their corresponding answers as question-answer data pairs, so that the question-answer data pairs can cover not only multiple dynamic scenes but also multiple motion attributes. Therefore, compared to existing spatial reasoning datasets that cover simple, controlled scenes and can only cover some simple questions, the spatial reasoning dataset generated by this embodiment of the disclosure based on target videos and multiple question-answer data pairs corresponding to the target videos covers multiple dynamic scenes and multiple motion attributes. Training a visual language model with this spatial reasoning dataset enables the visual language model to understand and reason about changes in spatial relationships between objects in dynamic scenes.

[0061] It is worth noting that some videos depict still objects, meaning there are no obviously moving objects in the video. It is impossible to generate 3D spatial trajectories for each object from such videos, nor can these videos be used to train the spatial reasoning ability of a visual language model. Therefore, it is necessary to filter out videos without moving objects beforehand to ensure that the final spatial reasoning dataset can be used to train the spatial reasoning ability of the visual language model.

[0062] In one possible implementation, multiple target videos are acquired, specifically a video dataset; target videos containing moving objects are then selected from the video dataset.

[0063] In this context, a video dataset refers to a collection containing multiple videos. Furthermore, these videos cover a wide range of scenarios, including simple, controlled scenes (such as "a ball rolling on a plane" or "blocks collapsing after being stacked" generated by the Unity engine), dynamic scenes involving multi-object interactions (such as "pedestrians, shopping carts, and goods moving and avoiding each other near supermarket shelves"), scenes depicting the dynamic changes of non-rigid objects (such as "fabric being stretched" or "liquid flowing"), and real-world outdoor scenes (such as "the dynamic intersection of vehicles, pedestrians, and bicycles at an intersection, and changes in spatial relationships affected by traffic lights and obstacles"). Based on this, the video dataset covers a diverse range of dynamic scenarios. Training a visual language model using such scenarios avoids limiting the spatial reasoning dataset to a finite number of specific scenes, allowing the visual language model to more fully understand the changing spatial relationships between objects in dynamic scenes.

[0064] Specifically, the video dataset can be the general video dataset Koala-36M [8], which contains a large number of videos, ensuring that the videos in the dataset cover a variety of dynamic scenes. Moreover, the Koala-36M [8] dataset usually includes not only the video itself, but also a text describing the overall events of the video and the objects contained in the video.

[0065] Understandably, a large number of videos in the video dataset lack obvious object motion, making it impossible to construct a valid spatial reasoning dataset based on such videos, thus hindering the training of the spatial reasoning capabilities of the visual language model. Therefore, it is necessary to filter the videos in the dataset to remove those without obvious object motion (e.g., static landscapes, static scenes with fixed shots), and select those with obvious object motion as target videos. Since a large number of videos without obvious motion consumes storage resources and prolongs the training time of the visual language model (e.g., the computational overhead of traversing invalid videos), filtering out videos without obvious motion effectively reduces redundant data and lowers computational and time costs.

[0066] In one possible implementation, target videos containing moving objects are selected from the video dataset. Specifically, for each video in the video dataset, multiple video frames are obtained; based on the multiple video frames, it is determined whether there is moving objects in the video; when there is moving objects in the video, the video is identified as the target video.

[0067] As mentioned above, the video consists of multiple video frames arranged in chronological order, with each video frame representing a scene from the target video at a specific moment. Based on this, refer to... Figure 5 Visual language models can be used to analyze multiple video frames to determine if there are visual changes (such as object position shifts, pose changes, or relative background displacement) between them. By analyzing these visual changes, the motion state of each object in the video can be accurately determined. If the visual language model outputs "no visual changes exist in the multiple video frames corresponding to the video," then it can be determined that no object is moving in the video, and this video can be directly discarded. If the visual language model outputs "visual changes exist in the multiple video frames corresponding to the video," then it can be determined that object movement exists in the video, and this video can be identified as the target video.

[0068] For example: if multiple video frames are like Figure 4A As shown, through Figure 4A It can be determined that there are obvious changes in the image of these multiple video frames, therefore it can be... Figure 4AThe video corresponding to multiple video frames of video B is determined as the target video. If multiple video frames of video B are as follows... Figure 5 As shown, through Figure 5 It can be determined that these multiple video frames do not change, therefore it can be confirmed that... Figure 5 Video B contains no moving objects, so video B is removed.

[0069] Specifically, the visual language model can be Gemini-2.5-Pro, which possesses excellent inter-frame difference analysis capabilities. It can not only identify large-scale motion (e.g., vehicles moving, pedestrians running) but also distinguish between obvious and subtle motion (e.g., leaves swaying slightly, objects shaking slightly). Furthermore, Gemini-2.5-Pro ​​can accurately filter videos with motion by using preset thresholds (e.g., inter-frame object displacement exceeding a pixel threshold, pose change exceeding an angle threshold), improving the accuracy of video selection. In some cases, these thresholds can be adjusted according to actual task requirements. For example, decreasing the pixel / angle threshold to retain more videos with subtle motion, or increasing the pixel / angle threshold to focus on videos with obvious motion, can adapt to different task needs.

[0070] Understandably, by analyzing multiple video frames using a visual language model, the motion state of various objects in the video can be accurately determined by utilizing the visual changes between the frames. Furthermore, when visual changes exist across multiple video frames, the video is identified as the target video; when no visual changes are observed, the video is discarded. This approach leverages visual changes to achieve precise video selection, ensuring the usability of the selected target videos.

[0071] It's worth noting that directly determining the presence of motion in a 3D paper image using a visual language model may lead to false positives due to subtle pixel variations. Therefore, to improve the accuracy of the filtering and ensure that all selected target videos exhibit motion, the following method can be used to filter videos in the dataset, removing those without obvious motion and selecting those with clear motion.

[0072] In one possible implementation, target videos containing object motion are filtered from the video dataset. Specifically, for each video in the video dataset, the corresponding video description text is obtained; the existence of a description of object motion in the video description text is detected; when a description of object motion exists in the video description text, the video is identified as the target video.

[0073] Among these, the video description text corresponding to the video refers to the text used to describe the time when the video takes place and the objects contained in the video; that is, the text describing the events that occur in the video. For example: refer to Figure 4A , Figure 4A The video description text corresponding to the video could be: "The vehicle in front of the camera, while accelerating, first veered to the right, then continued accelerating. After accelerating for a distance, it slowed down and veered to the left." (Refer to...) Figure 4B , Figure 4B The video description text could be: The vehicle that initially appeared on the left side of the camera's viewfinder veered to the left and then continued moving at a constant speed. The vehicle that initially appeared on the right side of the camera's viewfinder veered to the right while moving forward and then continued moving at a constant speed. Finally, the vehicles that initially appeared on the left and right sides of the camera's viewfinder drove in opposite directions.

[0074] Understandably, as mentioned above, to remove videos without obvious object movement and filter out videos with obvious object movement, a large language model can be used to analyze the video description text corresponding to each video in the video dataset. This analysis will determine whether the video description text contains descriptions of object movement. Figure 5 As shown. If the output of the large language model is "there is no description of object motion in the video description text", then the video corresponding to this video description text can be directly removed. If the output of the large language model is "there is a description of object motion in the video description text", then the video corresponding to this video description text can be identified as the target video.

[0075] Specifically, the large language model, pre-trained on massive amounts of text data, possesses extremely strong semantic understanding and contextual analysis capabilities. When faced with video description text, the large language model does not simply match words related to "motion," but can also infer whether objects are obviously moving through contextual information such as scenes and actions in the video description text. For example, if the video description text is "The water glass on the table is still by the window, and the leaves outside the window are not moving at all," the large language model can determine that this is a static scene, and there is no description of object movement in the video description text. If the video description text is as described above, "The vehicle in front of the camera first veers to the right while accelerating, then continues to accelerate, and after accelerating for a distance, it slows down and veers to the left," the large language model can determine that this is a dynamic scene, and there is a description of object movement in the video description text. Through the deep semantic parsing capabilities of the large language model, the false positives that are prone to occur when judging motion solely based on pixel changes can be significantly reduced, allowing the target video to fully reflect the changes in spatial relationships between objects in a dynamic scene.

[0076] It should be noted that, as mentioned above, the video dataset can be the general video dataset Koala-36M [8]. This is because the Koala-36M [8] dataset not only includes the video itself, but also a text describing the overall events and objects contained in the video (i.e., video description text). Therefore, after obtaining the video dataset, the video description text corresponding to each video can be analyzed directly through a large language model, without having to generate video description text based on the overall events and objects contained in the video. This improves the efficiency of identifying the target video.

[0077] Understandably, by using a large language model to detect whether there is a description of object movement in the video description text, the video is identified as the target video when there is a description of object movement in the video description text, and the video is removed when there is no description of object movement in the video description text. This realizes the deep semantic parsing capability of the large language model, which greatly reduces the false screening that is easy to occur when judging motion solely by pixel changes, and allows the target video to fully reflect the changes in spatial relationships between objects in a dynamic scene.

[0078] In one possible implementation, the three-dimensional spatial trajectory of each object in the target video is determined. Specifically, this can be achieved by extracting the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame; and then determining the three-dimensional spatial trajectory of each object in the target video based on the camera pose, point cloud distribution, object mask, and object pose.

[0079] In this context, the camera pose in each video frame refers to the set of the camera's position (spatial coordinates) and orientation (shooting direction and angle) in a 3D coordinate system for each frame. Camera pose determines not only the camera's location during video capture but also its orientation and angle at a given moment. The 3D coordinate system includes a world coordinate system and an object-local coordinate system. The world coordinate system is a fixed global coordinate system, such as one with the initial point of the target video as the origin, the ground as the XY plane, and the Z-axis perpendicular to the ground. The camera pose in the world coordinate system describes the camera's objective position and orientation in global space. The object-local coordinate system is a dynamic coordinate system with a specific object in the target video as its origin, such as the center of a moving vehicle as the origin, the ground as the XY plane, and the Z-axis perpendicular to the ground. The camera pose in the object-local coordinate system describes the camera's position and orientation relative to that object; for example, when the camera is fixed to a vehicle, its relative pose changes with the vehicle's movement.

[0080] In this context, the point cloud distribution for each video frame refers to the set of discrete points used to describe the scene and the three-dimensional spatial structure of objects within that video frame. The point cloud distribution is strictly bound to the video frame; each frame's point cloud distribution only describes the static three-dimensional scene at the corresponding moment in the video frame. The temporal combination of multiple consecutive video frames can reflect the dynamic changes in the scene. Specifically, each point cloud consists of a large number of three-dimensional discrete points (typically hundreds of thousands to millions), and each discrete point also contains the three-dimensional coordinates of its spatial location in a specified coordinate system. The point cloud distribution in a single video frame can completely cover all objects and scene backgrounds (e.g., roads, buildings, dynamic objects), and can reconstruct the three-dimensional shape of objects (e.g., vehicle outlines, pedestrian postures) and spatial relationships (e.g., relative distances between objects), unlike the planar projection information only found in 2D frames.

[0081] In this context, the object mask in each video frame refers to a binarized / multi-valued image used to accurately determine the pixel orientation of objects in each video frame. A binarized object mask is a pixel map with only two values, 0 and 1, and the same resolution as the video frame. "1" represents a pixel belonging to the target object, and "0" represents a pixel of the background or other objects. For example, the pixel corresponding to vehicle A in a video frame is labeled as 1, and the rest are labeled as 0. A multi-valued object mask is a pixel map with the same resolution as the video frame, used to assign different values ​​to different objects (e.g., vehicle A=1, vehicle B=2, background=0). Multi-valued object masks can simultaneously isolate multiple moving objects. Without extracting the object mask for each video frame, the pixels of the background and static objects would cause significant interference in the subsequent generation of 3D spatial trajectories. By extracting the object mask, the interference from the background and static objects can be eliminated, ensuring that subsequent processing focuses only on dynamic objects.

[0082] In this context, the object pose of each object in a video frame refers to the set of the object's spatial position and orientation (direction and spatial angle) in the corresponding 3D coordinate system for each video frame. The object pose reflects both the object's 3D coordinates at that moment in the video frame and its orientation and angle at that instant. For a single object, the object poses from multiple consecutive video frames can reconstruct the object's motion details in a dynamic scene, such as movement, turning, and translation. By extracting the object pose of each object in each video frame, the changes in position and orientation of each object over time can be clearly identified.

[0083] It should be noted that the camera pose, point cloud distribution, object pose, and object mask data for each video frame only describe the instantaneous state of an object in a single video frame. To clarify the laws governing object motion, it is necessary to reconstruct the complete motion process of each object in the target video. Therefore, after extracting the camera pose, point cloud distribution, object mask, and object pose of each object in the target video, the geometric feature data (i.e., camera pose, point cloud distribution, object pose, and object mask) contained in each video frame can be concatenated according to the temporal order of the video frames to determine the 3D spatial trajectory of each object. The 3D spatial trajectory simultaneously carries visual geometric features (such as position and displacement) and motion features (such as motion trends). Based on the 3D spatial trajectory, question-and-answer data pairs targeting multiple motion attributes can be quickly generated.

[0084] Understandably, by extracting the camera pose, point cloud distribution, object mask, and object pose of each object in each video frame from the target video, and determining the 3D spatial trajectory of each object in the target video based on the camera pose, point cloud distribution, object mask, and object pose, the static geometric feature data of each video frame is transformed into a continuous dynamic expression. This facilitates the subsequent generation of question-and-answer data that focuses only on the motion process of each object, thereby training the visual language model's ability to understand and reason about changes in spatial relationships between objects in dynamic scenes.

[0085] In one possible implementation, the camera pose of the camera capturing the target video in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame are extracted. Specifically, the category of each object in the target video can be determined based on the video description text corresponding to the target video; the target video is uniformly sampled to obtain a video frame sequence; based on the category of each object in the target video, the camera pose of the camera capturing the target video in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame are extracted from the video frame sequence.

[0086] In this context, the category of each object in the target video refers to the entity category label explicitly mentioned or implicitly stated in the video description text, possessing independent spatial form and motion trend. For example, if the video description text of the target video is "A busy intersection, pedestrians crossing the road, vehicles waiting for the green light," the object category of pedestrians can be determined as objects with obvious movement, the object category of vehicles as objects that may move, and the object category of the road as static background. Specifically, a large language model can be used to analyze the video description text corresponding to the target video to semantically classify each object and obtain its category.

[0087] Understandably, by determining the category of each object in the target video through the video description text corresponding to the target video, it is possible to identify the objects with obvious movement that need to be detected subsequently, and filter out static objects and static backgrounds that do not move, so as to avoid invalid image processing of static objects and static backgrounds during subsequent feature extraction.

[0088] In this context, a video frame sequence refers to a sequence of multiple video frames ordered chronologically; that is, a video frame sequence includes multiple consecutive video frames. Uniform sampling of the target video involves extracting a fixed number of video frames from the original video frame sequence at fixed time intervals, forming a temporally uniform video frame sequence of consistent length. By uniformly sampling each target video to obtain a video frame sequence, target videos of different durations and frame rates can be standardized into frame-level data of a unified format, facilitating subsequent batch extraction of geometric feature data such as camera pose, point cloud distribution, object pose, and object mask.

[0089] It is understandable that, because the video frames in a video frame sequence are equidistantly distributed on the time axis, by uniformly sampling the target video, these continuous video frames can stably capture the continuous process of object motion, avoiding the loss of motion details of objects due to dense or sparse sampling.

[0090] Furthermore, after determining the categories of each object in the target video and the video frame sequence, for each video frame in the video frame sequence, the object category (e.g., an object with obvious motion) can be used as a semantic cue word, and input along with the video frame into a model used to determine the object mask in the video frame. This allows for the accurate localization of each object corresponding to that category, and the output is the object mask in the video frame. Similarly, for each video frame in the video frame sequence, the object category can also be used as a semantic cue word, and input along with the video frame into a model used to determine the object pose of each object in the video frame. This allows for the accurate localization of each object corresponding to that category, and the output is the object pose of each object in the video frame.

[0091] Understandably, by identifying the categories of objects in the target video, the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame can be extracted from the video frame sequence. This allows for the clear extraction of objects with poses and object masks, avoiding ineffective processing of static and irrelevant objects and reducing the waste of computing power.

[0092] In one possible implementation, based on the category of each object in the target video, the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame are extracted from the video frame sequence. Specifically, a visual understanding model can be used to analyze the video frame sequence to obtain the camera pose and the point cloud distribution of each video frame; a mask extraction model can be used to analyze the video frame sequence and the object category to obtain the object mask of each video frame; and a pose analysis model can be used to analyze the video frame sequence to obtain the object pose of each object in each video frame.

[0093] It's important to note that without a known camera pose, the 3D coordinates of each point cloud will lack a reference (for example, the point cloud coordinates of the same object cannot be unified under different viewpoints). Without point clouds, the true 3D position of objects in the target video cannot be obtained; it remains at the 2D pixel level. Therefore, it's necessary to first analyze the video frame sequence using a visual understanding model to obtain the camera pose (the 3D position and orientation of the camera) and the point cloud distribution of each video frame (the point cloud of all objects and the background in the video frame), such as... Figure 6 As shown. Among them, the visual understanding model refers to the deep learning model used to recover three-dimensional spatial information (i.e., camera pose and point cloud distribution) from a 2D video frame sequence. In some cases, the visual understanding model can be the π3[9] model.

[0094] It's worth noting that because visual understanding models generate point clouds of the entire scene within video frames—meaning the point cloud distribution includes both the background and all objects—it's impossible to determine objects with significant motion based solely on the point cloud distribution. Therefore, object categories and video frame sequences can be input into a mask extraction model. This allows the model to generate pixel-level masks of objects with significant motion in each video frame based on their categories, such as... Figure 6 As shown. Simply put, it involves identifying which areas in a video frame contain objects with obvious movement and which are static backgrounds. In some cases, the mask extraction model can be the Grounded SAM2

[10] model.

[0095] Understandably, to determine the 3D spatial trajectory of each object, it is necessary not only to know its position but also its orientation to determine the direction in which it will move. Based on this, a sequence of video frames can be input into a pose analysis model to identify the orientation of each object (e.g., the direction a vehicle is facing, the facing direction of a pedestrian), and output the pose of each object (e.g., orientation angle, pose matrix). Figure 6As shown. In some cases, the pose analysis model can be the Orient Anything

[11] model. By extracting the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame from the video frame sequence, the instantaneous state of the object in each video frame can be determined, so as to restore the complete motion process of each object in the target video and obtain the three-dimensional spatial trajectory.

[0096] In one possible implementation, the 3D spatial trajectory of each object in the target video is determined based on camera pose, point cloud distribution, object mask, and object pose. Specifically, it can be based on the object mask in each video frame, filtering the point cloud of the object from the point cloud distribution of each video frame; for each object in each video frame, the position of the object in 3D space is determined based on the object's point cloud; and the positions of each object in 3D space in each video frame are sequentially connected according to the time order of the video frames to obtain the 3D spatial trajectory of each object in the target video.

[0097] In this context, the point cloud of an object refers to the discrete set of points describing the three-dimensional spatial structure of an object in a video frame. Because the point cloud distribution includes both static background and all objects, it's impossible to determine objects with significant motion based solely on the point cloud distribution. However, the object mask in each video frame is a pixel-level mask of objects with significant motion. Therefore, we can use the object masks in each video frame to filter out the point clouds of objects with significant motion from the point cloud distribution of each frame. This effectively prevents background point clouds (such as ground and buildings) and other static object point clouds from being mixed into the subsequent calculation of 3D spatial trajectories.

[0098] Understandably, calculating a 3D spatial trajectory requires knowing the object's location at each moment, and each discrete 3D point in the point cloud contains the 3D coordinates of that point's spatial location in a specified coordinate system. Therefore, for a single object, its position in 3D space at the corresponding moment in each video frame can be determined based on the object's point cloud in each video frame. Specifically, regardless of the object's category or the complexity of its shape (e.g., irregular shape), a unified approach can be adopted to calculate 3D positions: "Based on the object mask in each video frame, filter out the object's point cloud from the point cloud distribution of each video frame; for each object in each video frame, determine the object's position in 3D space based on the object's point cloud." This avoids inconsistencies in calculation logic caused by differences in object shape (e.g., for some conical objects, the vertex is used as the object's position in 3D space, while for some spherical objects, the center is used), unifying the calculation method for the positions of different objects in 3D space in different target videos, and adapting to the batch processing of large-scale video data.

[0099] Understandably, the previous step only yielded a static, instantaneous state of the objects' positions in 3D space within each video frame. To reconstruct the complete motion of each object in the target video, the positions of the objects in each video frame can be concatenated according to the temporal order of the video frames to obtain the 3D spatial trajectories of each object in the target video (including timestamps and the sequence of object positions in 3D space). This approach completely covers the entire motion process of the objects from the beginning to the end of the target video, ensuring that the motion attributes of any time period are not missed in the 3D spatial trajectory, thus ensuring the continuity of the 3D spatial trajectory.

[0100] In one possible implementation, the position of an object in three-dimensional space is determined based on the object's point cloud. Specifically, this can be achieved by obtaining the three-dimensional spatial coordinates of each point in the object's point cloud and using the average value of the three-dimensional spatial coordinates of each point as the position of the object in three-dimensional space.

[0101] Understandably, the 3D spatial coordinates of each point in an object's point cloud refer to the coordinates of discrete points describing the object's 3D spatial structure in a 3D coordinate system. Because an object's point cloud contains a large number of discrete 3D points, these points may exhibit deviations in their 3D spatial coordinates due to shooting blur or noise in the visual understanding model. Therefore, the average of the 3D coordinates of all point clouds within a single video frame can be taken as the object's position in 3D space. In this case, even if some point clouds in a single video frame have positional deviations due to shooting blur or noise, these deviations will be offset by the statistical properties of the overall point cloud, improving the accuracy of the final obtained position of the object in 3D space.

[0102] In one possible implementation, questions and corresponding answers are generated based on a three-dimensional spatial trajectory, targeting multiple motion attributes. Specifically, for each motion attribute, a preset template is filled based on the motion attribute to obtain a question targeting the motion attribute; and the question targeting the motion attribute is calculated based on the three-dimensional spatial trajectory to obtain the corresponding answer.

[0103] The preset template refers to a pre-defined, reusable question structure framework. Preset templates reserve some empty spaces, which can be filled using motion attributes (such as distance, speed, pose, etc.). Specifically, generally, each motion attribute corresponds to a preset template, which typically includes a time constraint, the motion attribute, and the question format. For example, the preset template framework for the motion attribute of direction could be: "[Time constraint], the direction of [object A] and [object B] [Question type (how it changes)]?".

[0104] It should be noted that, for a single motion attribute, filling a preset template based on the motion attribute means: determining the time range based on the playback time of the target video, determining the object the question targets based on the objects contained in the target video, and using the motion attribute, time range, and the object the question targets to fill the empty spaces in the preset template. (See reference...) Figure 7 If the preset template is "[Time Limit], Direction of [Object A] and [Object B] [Question Type (How Does It Change)]?", with a time range of 2 to 4 seconds, and the object A being the camera and the object B being the car, and the motion attribute being direction, then by filling the empty spaces in the preset template with the motion attribute, the question targeting the motion attribute can be obtained as "Between 2 and 4 seconds, from the camera's perspective, how does the direction of the camera relative to the car located within the bounding box (432, 180, 461, 192) change?". By filling the preset template based on motion attributes, a large number of questions can be generated in batches to provide sufficient training samples for subsequent visual language models, compensating for the low efficiency of manual annotation of training samples.

[0105] Understandably, since questions asking about motion attributes are derived by filling templates with random motion attributes, the corresponding answers are not yet clear. Therefore, we can calculate the answers based on 3D spatial trajectories. Because a 3D spatial trajectory contains the object's 3D coordinates and pose at various moments, and the position and pose of each object are the basis for calculating all motion attributes, the answer to the question can be quickly determined based on the 3D spatial trajectory. This allows the question and its corresponding answer to focus on multiple motion attributes, thereby accurately training the visual language model's dynamic spatial cognition and enabling the model to understand the changing spatial relationships between objects in a dynamic scene.

[0106] In one possible implementation, a preset template is populated based on motion attributes to obtain questions that address the motion attributes. Specifically, the preset template is populated based on motion attributes to obtain question text that addresses the motion attributes. Multiple question options corresponding to the motion attributes are determined based on the motion attributes and a preset set of question options.

[0107] The questions targeting motion attributes include question text and multiple question options. The question text refers to a short passage asking about the motion attribute, without any options. The step of filling a preset template with information about the motion attribute to obtain the question text specifically involves: for each target video, organizing the 3D spatial trajectory into structured data according to the structure of timestamp-object-values ​​of various motion attributes. For example: timestamp = 8 seconds, vehicle A = position (X1, Y1, Z1), vehicle B = position (X2, Y2, Z2), the distance between vehicle A and vehicle B, the speed of vehicle A, and the speed of vehicle B. After obtaining this structured data, a timestamp, object, and a specific motion attribute can be randomly selected from the structured data and filled into a preset template to generate the question text targeting that motion attribute. For example: Select timestamp = 8 seconds (the time interval between adjacent video frames is 1 second), object A = vehicle A, object B = vehicle B, motion attribute = distance, preset template question type = how much, and the final generated question text is "What is the 3D distance between vehicle A and vehicle B in the 8th video frame?" The question option set includes multiple question options corresponding to each motion attribute. Question options refer to the candidate options following the question text for selection. As described above, this embodiment provides the following question options for each motion attribute: For distance and speed, the following options are provided: (1) Remains essentially unchanged; (2) Remains essentially unchanged then increases; (3) Remains essentially unchanged then decreases; (4) Gradually increases; (5) Gradually decreases. For direction, pose, and predicted direction, the following options are provided: (1) Forward / Backward; (2) Left / Right; (3) Up / Down. For speed comparison (e.g., which is faster, object A or object B), the following options are provided: (1) Both are the same; (2) The former is faster; (3) The latter is faster. In practice, the question options can be reduced or modified based on the three-dimensional spatial trajectory so that the multiple question options corresponding to the question text contain the correct answer to the question.

[0108] Understandably, by filling a preset template with motion attributes, questions that target the motion attributes are obtained. Then, based on the three-dimensional spatial trajectory, the questions that target the motion attributes are calculated to obtain the corresponding answers. A large number of questions can be generated in batches to provide sufficient training samples for subsequent visual language models, thus making up for the low efficiency of manual annotation of training samples.

[0109] In one possible implementation, the preset template is filled with motion attributes to obtain question text that asks about motion attributes. Specifically, when the preset template is an absolute viewpoint template, the static reference frame, the target object, and the time range of the question are determined, and the absolute viewpoint template is filled with the static reference frame, target object, time range, and motion attributes to obtain question text that asks about motion attributes. When the preset template is a relative viewpoint template, the dynamic reference frame, the target object, and the time range of the question are determined, and the relative viewpoint template is filled with the dynamic reference frame, target object, time range, and motion attributes to obtain question text that asks about motion attributes.

[0110] The preset templates include absolute view templates and relative view templates.

[0111] Among them, the absolute perspective template refers to a question template that uses a static reference frame (such as a reference frame with a stationary object in the target video as the origin) as the observation benchmark and asks questions about motion attributes. For example, the absolute perspective template could be: "From the perspective of [static reference frame], [within time limit], what are the [motion attributes] of [object A] [question type (how does it change)]?"

[0112] Understandably, when the preset template is an absolute perspective template, after determining the static reference frame, time range, target object, and motion attributes of the question, the static reference frame, time range, target object, and motion attributes can be used to fill the absolute perspective template to obtain the question text that asks about motion attributes. For example, as mentioned above, when the absolute perspective template is "From the perspective of [static reference frame], [time limit], [object A]'s [motion attribute] [question type (how it changes)]?", the static reference frame is the camera (the camera did not move during the capture of the target video), the time range is from the 8th to the 12th video frame, the target object is vehicle A, and the motion attribute is speed, after filling the absolute perspective template based on the static reference frame, target object, time range, and motion attribute, the resulting question text is "From the perspective of the camera, how does the speed of vehicle A change from the 8th to the 12th video frame?" This not only avoids the reasoning bias of the visual language model caused by perspective ambiguity, ensuring the uniqueness and accuracy of the answer, but also allows the visual language model to learn and master the motion laws of objects under absolute perspective in dynamic scenes.

[0113] Among them, the relative perspective template refers to a question template that uses a dynamic reference frame (such as a reference frame with an object in the target video that is clearly moving as the origin) as the observation benchmark and asks questions about motion attributes. For example, if vehicle A in the target video is clearly moving, the relative perspective template could be: "From the perspective of [dynamic reference frame], [within time limits], what are the [motion attributes] of [object A] [question type (how does it change)]?"

[0114] Understandably, when the preset template is a relative perspective template, after determining the dynamic reference frame, time range, target object, and motion attributes of the question, the relative perspective template can be filled with the dynamic reference frame, time range, target object, and motion attributes to obtain question text that asks about motion attributes. For example, as mentioned above, when the relative perspective template is "From the perspective of [dynamic reference frame], [time limit], [object A]'s [motion attribute] [question type (how it changes)]?", the dynamic reference frame is vehicle A (vehicle A is moving at a constant speed in the target video), the time range is from the 1st to the 4th video frame, the target object is vehicle B, and the motion attribute is speed, after filling the relative perspective template with the dynamic reference frame, target object, time range, and motion attributes, the resulting question text is "From the perspective of vehicle A, how does the speed of vehicle B change from the 1st to the 4th video frame?" This allows the visual language model to learn and master the motion patterns of objects in a dynamic scene from a relative perspective.

[0115] In one possible implementation, the question posed regarding motion attributes is calculated based on the three-dimensional spatial trajectory to obtain the answer corresponding to the question. Specifically, this can be achieved by transforming the three-dimensional spatial trajectories of each object to the reference frame targeted by the question, obtaining the transformed three-dimensional spatial trajectories of each object; determining multiple consecutive video frames corresponding to the time range targeted by the question; for adjacent video frames, calculating the changes in motion attributes within adjacent video frames based on the transformed three-dimensional spatial trajectories to obtain sub-answers to the question; and integrating multiple sub-answers to obtain the answer corresponding to the question.

[0116] It is worth noting that the relative position and orientation of objects will change from different perspectives (for example, the orientation of an object may change when the camera moves). Therefore, it is necessary to transform the three-dimensional spatial trajectories of each object to the reference frame applicable to the problem to avoid incorrect answers due to different reference frames for the three-dimensional spatial trajectories of each object.

[0117] Understandably, determining multiple consecutive video frames corresponding to the time range addressed by the question means identifying consecutive video frames uniformly sampled within that time range of the target video. When an object exhibits significant motion, its position and pose may change in each video frame of the target video. To clarify these changes, for the transformed 3D spatial trajectory of the object, it is necessary to calculate the changes in motion attributes between adjacent video frames to determine these minute changes in motion attributes, i.e., the sub-answers to the question. For example, if the question is "How does the distance between the boy and the dog change between 3s and 16s?", and 3s-16s corresponds to 10 consecutive video frames, for the transformed 3D spatial trajectory of the boy and the dog, the sub-answer calculated for the first to second video frame could be that the distance remains constant; the sub-answer calculated for the second to third video frame could also be that the distance remains constant; the sub-answer calculated for the third to fourth video frame could be that the distance increases; and the sub-answer calculated for the fourth to tenth video frame could be that the distance continuously increases.

[0118] Understandably, after identifying the minute changes in motion attributes between adjacent video frames, the process for handling these minute changes is to merge consecutive, repetitive minute changes. The final answer obtained is the result of merging the sub-answers of the question. For example, if the question is "How does the distance between the boy and the dog change between 3s and 16s?", after integrating the multiple sub-answers mentioned above, the answer corresponding to the question can be "remains basically unchanged, then increases." By integrating multiple sub-answers to obtain the answer corresponding to the question, redundant information from repeated video frames can be filtered out, highlighting the core trends in motion attribute changes, enabling the visual language model to understand and infer changes in spatial relationships between objects in a dynamic scene.

[0119] In one possible implementation, questions and corresponding answers are generated based on a three-dimensional spatial trajectory, which involve analyzing the three-dimensional spatial trajectory using a question generation model to generate questions and corresponding answers that address multiple motion attributes.

[0120] Among them, the question generation model refers to a deep learning model used to generate questions and answers requiring spatial reasoning based on the three-dimensional spatial trajectory of an object. Traditional question-answer pair generation typically relies on manually pre-setting motion attributes and question formats (e.g., multiple choice, true / false questions). This results in a monotonous format of question-answer pairs, which, while conforming to human thought processes, fails to reflect real-world dynamic scenarios. Therefore, the three-dimensional spatial trajectory can be converted into a structured text format (temporally presenting the object's position and pose information) that the question generation model can understand. The structured text format of the three-dimensional spatial trajectory is then input into the question generation model, such as... Figure 7 As shown.

[0121] Understandably, after receiving a structured text-based 3D spatial trajectory, the question generation model transforms it into a semantic motion description. Then, based on this semantic description, it automatically mines noteworthy motion attributes and, based on these attributes, autonomously designs diverse questions, calculating the answers to these questions using the 3D spatial trajectory. Traditional question-and-answer pairs typically use fixed formats like multiple-choice or true / false questions, while the question generation model can generate diverse question-and-answer pairs such as interrogative sentences, calculation problems, comparison questions, prediction questions, and analysis questions, more closely resembling human questioning habits in real-world scenarios. This approach improves the adaptability of subsequent visual language models to real-world situations.

[0122] In one possible implementation, a spatial reasoning dataset is generated based on the target video and multiple question-and-answer data pairs corresponding to the target video. Specifically, a first number of target videos and their corresponding question-and-answer data pairs can be used as a training dataset, and a second number of target videos and their corresponding question-and-answer data pairs can be used as an evaluation dataset, where the second number is less than the first number. The training dataset and the evaluation dataset are then integrated to obtain the spatial reasoning dataset.

[0123] The spatial reasoning dataset comprises a training dataset and an evaluation dataset. The training dataset is a set of samples used to train the spatial reasoning ability of the visual language model. It contains a first set of target videos and corresponding question-and-answer pairs. The evaluation dataset is a set of samples used to test the spatial reasoning ability of the visual language model. It contains a second set of target videos and corresponding question-and-answer pairs. The question-and-answer pairs corresponding to the target videos refer to all question-and-answer pairs generated for each object and each motion attribute in the target videos. In other words, the question-and-answer pairs corresponding to the target videos cover all motion attributes and all objects in the target videos.

[0124] Understandably, the first and second numbers can be determined based on the specific number of target videos. The sum of the first and second numbers represents the number of target videos. To ensure that the visual language model can fully understand and infer the spatial relationship changes between objects in a dynamic scene, the first number is much larger than the second number. For example, with 5150 target videos, 5000 target videos and their corresponding question-and-answer pairs can be used as the training dataset, and 150 target videos and their corresponding question-and-answer pairs can be used as the evaluation dataset. By setting the first number to be much larger than the second number, it can be ensured that the visual language model can fully understand and infer the spatial relationship changes between objects in a dynamic scene.

[0125] In one possible implementation, after generating a spatial reasoning dataset based on the target video and multiple question-and-answer data pairs corresponding to the target video, the visual language model can be trained based on the spatial reasoning dataset.

[0126] Understandably, after training a visual language model on a spatial reasoning dataset, performance tests can be conducted on the visual language model when performing different types of sub-tasks to obtain results such as... Figure 8 The table shown. Figure 8 In the diagram, each column represents a different type of subtask, such as calculating absolute distance, calculating absolute direction, and calculating absolute pose. Each row represents a different model, such as GPT-4o, Qwen2.5-VL-7B, and VLM-3B. The value in each cell represents the model's performance score when performing that subtask; a higher value indicates better model performance. Figure 8 As can be seen, the visual language model trained on the spatial reasoning dataset of this disclosure outperforms other models (including closed-source, open-source, and models specifically designed for video understanding or spatial reasoning) in performing various sub-tasks of dynamic spatial reasoning. This intuitively demonstrates the effectiveness of the spatial reasoning dataset constructed in this disclosure.

[0127] In one possible implementation, the visual language model reasones about the problem by: receiving a spatial reasoning problem and a spatial reasoning video; converting the spatial reasoning problem into a text feature vector and the spatial reasoning video into a three-dimensional geometric feature vector; performing semantic compression on the text feature vector to obtain a compressed text feature vector; extracting a target three-dimensional geometric feature vector from the three-dimensional geometric feature vector that is semantically related to the compressed text feature vector; and performing fusion reasoning based on the text feature vector and the target three-dimensional geometric feature vector to obtain a solution corresponding to the spatial reasoning problem.

[0128] Among them, spatial reasoning videos refer to videos input by users that require spatial reasoning, and spatial reasoning questions refer to questions posed by users to the spatial reasoning videos. For example... Figure 9 As shown, after receiving the spatial reasoning question and the spatial reasoning video, the visual language model first transforms the spatial reasoning text into a text feature vector, which is the smallest semantic unit of the spatial reasoning text, through a text segmenter. Then, it transforms the spatial reasoning model into a three-dimensional geometric feature vector, which is the core representation unit obtained after discretizing and compacting three-dimensional spatial information (such as object shape, scene structure, trajectory, etc.), such as object position, point cloud, and pose.

[0129] It is worth noting that existing visual language models rely on cross-attention between text feature vectors transformed from spatial reasoning problems and 3D geometric feature vectors (such as the 3D coordinate features of each point in a point cloud distribution, or global features in a video) to make the text feature vectors focus on the 3D geometric feature vectors, allowing the 3D geometric feature vectors to supplement the spatial dimension information of the text feature vectors, or by directly adding the 3D geometric feature vectors to the visual feature vectors transformed from spatial reasoning videos (allowing the 3D geometric feature vectors to be directly superimposed on the visual feature vectors) to enhance spatial reasoning capabilities. However, because the number of 3D geometric feature vectors is large, and for general videos, 3D geometric feature vectors often contain irrelevant and noisy features, if the above two methods are used to enhance the visual understanding ability of visual language models, the 3D geometric feature vectors will crowd out the semantic weights of the visual feature space, causing the visual feature space of the visual language model to change from a general semantic space to a spatial semantic space, resulting in a serious decline in the general visual understanding ability of the visual language model and affecting its application in general scenarios. Therefore, after determining the text feature vectors, learnable queries can be combined with a semantic compression layer to semantically compress the text feature vectors. This allows the semantic compression layer to more accurately extract the core semantics of spatial reasoning problems, rather than relying solely on the text feature vectors. This approach makes the subsequent fused knowledge more accurate and explicit, avoiding interference from irrelevant and noisy information on the general capabilities of the visual language model.

[0130] Understandably, referring to Figure 9After semantically compressing the text feature vectors to obtain compressed text feature vectors (i.e., the core semantics of the spatial reasoning problem), a relevant geometric filtering layer is needed to selectively extract 3D geometric feature vectors. Only 3D geometric feature vectors that are relevant to the compressed text feature vectors are extracted as target 3D geometric feature vectors related to the spatial reasoning problem. Specifically, the relevant geometric filtering layer operates as follows: using the compressed text feature vectors as query keywords, and performing attention filtering on all 3D geometric feature vectors based on the query keywords, retaining only 3D geometric feature vectors that are semantically strongly related to the query keywords. Finally, a fixed number of 3D geometric feature vectors are output as target 3D geometric feature vectors. For example, 16 target 3D geometric feature vectors are selected from 1024 3D geometric feature vectors. By doing so, the 3D geometric feature vectors are made semantically strongly relevant to the spatial reasoning problem, effectively reducing redundant 3D geometric feature vectors and solving the problem of all 3D geometric feature vectors polluting the visual feature space of the visual language model.

[0131] Understandably, after determining the target's 3D geometric feature vector, both the target's 3D geometric feature vector and the text feature vector can be input into the fusion inference layer. This allows the fusion inference layer to fuse the target's 3D geometric feature vector and the text feature vector, completing spatial inference for the dynamic scene and outputting the solution corresponding to the spatial inference problem. By doing so, interference from irrelevant and noisy features in the integration of the 3D geometric feature vector and text feature vector can be avoided, ensuring that the 3D geometric feature vector fused with the text feature vector focuses on the spatial inference problem, thus improving the accuracy of the solution corresponding to the spatial inference problem.

[0132] In one possible implementation, the spatial reasoning video is converted into a three-dimensional geometric feature vector, specifically, the spatial reasoning video is converted into a visual feature vector; based on the visual feature vector, the three-dimensional geometric feature vector in the spatial reasoning video is extracted.

[0133] It is understandable that a visual feature vector refers to the basic representation unit obtained after segmenting and feature encoding a spatial reasoning video, and that the visual feature vector carries both visual and spatial features, such as shape, position, and color. (Refer to...) Figure 9After converting spatial reasoning videos into visual feature vectors using a video segmenter, these vectors can be input into a 3D base model. The 3D base model can then extract the corresponding 3D geometric features from the visual feature vectors, such as the position of objects in 3D space within each video frame, the point cloud distribution in each frame, and the camera pose in each frame. By converting spatial reasoning videos into visual feature vectors and extracting 3D geometric feature vectors from them, the visual language model can shift its focus from two-dimensional visual and spatial features to three-dimensional geometric features (such as the 3D coordinates, volume, and trajectory of objects). This facilitates the visual language model's understanding of spatial relationships between objects in dynamic scenes and enables spatial reasoning.

[0134] In one possible implementation, a solution corresponding to the spatial reasoning problem is obtained by fusing text feature vectors and target 3D geometric feature vectors. Specifically, this can be achieved by fusing text feature vectors, visual feature vectors, and target 3D geometric feature vectors to obtain a solution corresponding to the spatial reasoning problem.

[0135] Understandably, referring to Figure 9 After determining the target's 3D geometric feature vectors, the target's 3D geometric feature vectors, text feature vectors, and visual feature vectors are input together into the fusion inference layer. This allows the fusion inference layer to fuse these three feature vectors, completing spatial reasoning for the dynamic scene and outputting a solution corresponding to the spatial reasoning problem. By doing so, the visual language model can focus on the spatial reasoning problem, collaboratively understanding the linguistic semantics of the spatial reasoning problem, 2D visual and spatial features, and 3D geometric features. This avoids interference from a large number of 3D geometric feature vectors in the fusion processing, improving the accuracy of the solution corresponding to the spatial reasoning problem.

[0136] The following details the complete process of constructing the spatial reasoning dataset.

[0137] Reference Figure 10 , Figure 10 This is a schematic diagram of an optional architecture for a method of constructing a spatial reasoning dataset provided in an embodiment of this disclosure.

[0138] Step 1001: Obtain the video dataset, which includes multiple videos.

[0139] Step 1002: Filter out target videos containing object motion based on video description text. Specifically, this may include the following steps: For each video in the video dataset, obtain the corresponding video description text, which describes the events occurring in the video; detect whether the video description text contains a description of object motion, and if so, identify the video as a target video.

[0140] Step 1003: Filter out target videos containing moving objects based on multiple video frames. Specifically, this may include the following steps: For each video in the video dataset, obtain multiple video frames; based on the multiple video frames, determine whether there is moving objects in the video; when moving objects are present in the video, identify the video as the target video.

[0141] Step 1004: Extract camera pose, point cloud distribution, object mask, and object pose. Specifically, this may include the following steps: determining the category of each object in the target video based on the video description text corresponding to the target video; uniformly sampling the target video to obtain a video frame sequence, which includes multiple consecutive video frames; analyzing the video frame sequence using a visual understanding model to obtain the camera pose of the shooting camera and the point cloud distribution of each video frame; analyzing the video frame sequence and object categories using a mask extraction model to obtain the object mask for each video frame; and analyzing the video frame sequence using a pose analysis model to obtain the object pose of each object in each video frame.

[0142] Step 1005: Determine the 3D spatial trajectory. This may include the following steps: Based on the object mask in each video frame, filter out the object's point cloud from the point cloud distribution of each video frame; obtain the 3D spatial coordinates of each point in the object's point cloud; take the average of the 3D spatial coordinates of each point as the object's position in 3D space; and sequentially connect the positions of each object in 3D space in each video frame according to the time sequence of the video frames to obtain the 3D spatial trajectory of each object in the target video.

[0143] Step 1006: Fill in the preset template to obtain questions and corresponding answers regarding motion attributes. Specifically, this may include the following steps: When the preset template is an absolute viewpoint template, determine the static reference frame, the target object, and the time range for which the question pertains, and fill in the absolute viewpoint template based on the static reference frame, target object, time range, and motion attributes to obtain the question text regarding motion attributes; When the preset template is a relative viewpoint template, determine the dynamic reference frame, target object, and time range for which the question pertains, and fill in the relative viewpoint template based on the dynamic reference frame, target object, time range, and motion attributes to obtain the question text regarding motion attributes; Based on the motion attributes and a preset question option set, determine multiple question options corresponding to the motion attributes, wherein the question option set includes multiple question options corresponding to each motion attribute. The three-dimensional spatial trajectories of each object are transformed into the reference frame applicable to the problem to obtain the transformed three-dimensional spatial trajectories of each object; multiple consecutive video frames corresponding to the time range applicable to the problem are determined; for adjacent video frames, the changes in motion attributes within the adjacent video frames are calculated based on the transformed three-dimensional spatial trajectories to obtain sub-answers to the problem; multiple sub-answers to the problem are integrated to obtain the answer corresponding to the problem.

[0144] Step 1007: Analyze the three-dimensional spatial trajectory using a question generation model to generate questions that address multiple motion attributes and corresponding answers.

[0145] Step 1008: Collect the question and the corresponding answer as a question-answer data pair.

[0146] Step 1009: Generate a spatial reasoning dataset. Specifically, this may include the following steps: using a first number of target videos and their corresponding question-and-answer pairs as a training dataset; using a second number of target videos and their corresponding question-and-answer pairs as an evaluation dataset, where the second number is less than the first number; and integrating the training and evaluation datasets to obtain the spatial reasoning dataset.

[0147] Step 1010: Train the visual language model. This may specifically include the following steps: train the visual language model based on a spatial reasoning dataset.

[0148] It is worth noting that the visual language model reasones about the problem in the following way: receiving a spatial reasoning problem and a spatial reasoning video; converting the spatial reasoning problem into a text feature vector and the spatial reasoning video into a visual feature vector; extracting three-dimensional geometric feature vectors from the spatial reasoning video based on the visual feature vectors; semantically compressing the text feature vectors to obtain compressed text feature vectors; extracting target three-dimensional geometric feature vectors that are semantically related to the compressed text feature vectors from the three-dimensional geometric feature vectors; and performing fusion reasoning based on the text feature vectors, visual feature vectors, and target three-dimensional geometric feature vectors to obtain the solution corresponding to the spatial reasoning problem.

[0149] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0150] Reference Figure 11 , Figure 11 This is an optional structural diagram of a spatial reasoning dataset construction apparatus provided in an embodiment of the present disclosure. The spatial reasoning dataset construction apparatus 1100 includes: The acquisition module 1101 is used to acquire multiple target videos, wherein the target videos are videos in which objects are moving; The determination module 1102 is used to determine the three-dimensional spatial trajectory of each object in the target video for each target video. The first generation module 1103 is used to generate questions and corresponding answers based on a three-dimensional spatial trajectory, and to use the questions and corresponding answers as question-answer data pairs. The motion attributes indicate the attributes that the object has changing in three-dimensional space. The second generation module 1104 is used to generate a spatial reasoning dataset based on the target video and multiple question-and-answer data pairs corresponding to the target video.

[0151] Furthermore, the aforementioned determining module 1102 is specifically used for: Extract the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame from the target video. The 3D spatial trajectory of each object in the target video is determined based on camera pose, point cloud distribution, object mask, and object pose.

[0152] Furthermore, the aforementioned determining module 1102 is specifically used for: The category of each object in the target video is determined based on the video description text corresponding to the target video; The target video is uniformly sampled to obtain a video frame sequence, which includes multiple consecutive video frames. Based on the categories of each object in the target video, the camera pose of the camera in each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame are extracted from the video frame sequence.

[0153] Furthermore, the aforementioned determining module 1102 is specifically used for: By using a visual understanding model to analyze the video frame sequence, the camera pose of the shooting camera and the point cloud distribution of each video frame are obtained; The mask extraction model is used to analyze the video frame sequence and the object category to obtain the object mask for each video frame; The pose analysis model is used to analyze the video frame sequence to obtain the object pose of each object in each video frame.

[0154] Furthermore, the aforementioned determining module 1102 is specifically used for: Based on the object mask in each video frame, the point cloud of the object is filtered out from the point cloud distribution of each video frame. For each object in each video frame, the position of the object in three-dimensional space is determined based on the object's point cloud. By sequentially connecting the positions of each object in the three-dimensional space according to the time sequence of the video frames, the three-dimensional spatial trajectory of each object in the target video is obtained.

[0155] Furthermore, the aforementioned determining module 1102 is specifically used for: Obtain the three-dimensional spatial coordinates of each point in the point cloud of an object; The average value of the three-dimensional spatial coordinates of each point is taken as the position of the object in three-dimensional space.

[0156] Furthermore, the aforementioned first generation module 1103 is specifically used for: For each motion attribute, a preset template is populated based on the motion attribute to generate questions that address the motion attribute. The system calculates the answer to a question based on a three-dimensional spatial trajectory, targeting the motion properties of the question.

[0157] Furthermore, questions targeting motion attributes include question text and multiple question options; The first generation module 1103 mentioned above is specifically used for: The preset template is filled with information based on motion attributes to obtain question text that addresses motion attributes. Based on motion attributes and a preset set of question options, multiple question options corresponding to each motion attribute are determined. The question option set includes multiple question options corresponding to each motion attribute.

[0158] Furthermore, the preset templates include absolute viewpoint templates and relative viewpoint templates. The absolute viewpoint template is a question template that uses a static reference frame as the observation reference and asks questions about motion attributes. The relative viewpoint template is a question template that uses a dynamic reference frame as the observation reference and asks questions about motion attributes. The first generation module 1103 mentioned above is specifically used for: When the preset template is an absolute perspective template, the static reference frame, the target object, and the time range of the question are determined. The absolute perspective template is then filled based on the static reference frame, the target object, the time range, and the motion attributes to obtain the question text that asks about the motion attributes. When the preset template is a relative perspective template, the dynamic reference frame, the target object, and the time range of the question are determined. The relative perspective template is then filled based on the dynamic reference frame, the target object, the time range, and the motion attributes to obtain the question text that asks about the motion attributes.

[0159] Furthermore, the aforementioned first generation module 1103 is specifically used for: Transform the three-dimensional spatial trajectories of each object into the reference frame applicable to the problem to obtain the transformed three-dimensional spatial trajectories of each object. Identify multiple consecutive video frames that correspond to the time range addressed in the problem; For adjacent video frames, the changes in motion attributes within adjacent video frames are calculated based on the transformed 3D spatial trajectory to obtain sub-answers to the problem; By integrating multiple sub-answers to a question, the answer corresponding to the question is obtained.

[0160] Furthermore, the aforementioned first generation module 1103 is specifically used for: A question generation model is used to analyze 3D spatial trajectories to generate questions that address multiple motion attributes and corresponding answers.

[0161] Furthermore, the aforementioned acquisition module 1101 is specifically used for: Obtain the video dataset, which includes multiple videos; Filter out target videos from the video dataset that contain moving objects.

[0162] Furthermore, the aforementioned acquisition module 1101 is specifically used for: For each video in the video dataset, obtain the corresponding video description text, which is text describing the events that occur in the video; The system detects whether there is a description of object movement in the video description text. If there is a description of object movement in the video description text, the video is identified as the target video.

[0163] Furthermore, the aforementioned acquisition module 1101 is specifically used for: For each video in the video dataset, obtain multiple video frames of the video; Based on multiple video frames, it is determined whether there is motion in the video. When there is motion in the video, the video is identified as the target video.

[0164] Furthermore, the aforementioned second generation module 1104 is specifically used for: The first number of target videos and the question-and-answer data pairs corresponding to the target videos are used as the training dataset, and the second number of target videos and the question-and-answer data pairs corresponding to the target videos are used as the evaluation dataset, wherein the second number is less than the first number. The training dataset and the evaluation dataset are integrated to obtain the spatial reasoning dataset.

[0165] Furthermore, the apparatus for constructing the spatial reasoning dataset also includes a training module (not shown in the figure), which is specifically used for: The visual language model is trained based on the spatial reasoning dataset; Visual language models reason about problems in the following ways: Receive spatial reasoning questions and videos; The spatial reasoning problem is transformed into a text feature vector, and the spatial reasoning video is transformed into a three-dimensional geometric feature vector. Semantic compression is performed on the text feature vectors to obtain compressed text feature vectors; Extract target 3D geometric feature vectors that are semantically related to the compressed text feature vectors from 3D geometric feature vectors; Based on the fusion reasoning of text feature vectors and target 3D geometric feature vectors, a solution corresponding to the spatial reasoning problem is obtained.

[0166] Furthermore, the aforementioned training module is specifically used for: Convert spatial reasoning video into visual feature vectors; Based on visual feature vectors, extract three-dimensional geometric feature vectors from spatial reasoning videos; Based on the fusion reasoning of text feature vectors and target 3D geometric feature vectors, solutions corresponding to spatial reasoning problems are obtained, including: Based on the fusion reasoning of text feature vectors, visual feature vectors, and target 3D geometric feature vectors, a solution corresponding to the spatial reasoning problem is obtained.

[0167] The aforementioned spatial reasoning dataset construction apparatus 1100 and spatial reasoning dataset construction method are based on the same inventive concept. By acquiring multiple target videos containing moving objects, a massive amount of effective videos covering various dynamic scenes can be obtained, not just simple, controlled scenes. Then, for each target video, the spatial relationship changes between objects in the target video are clarified by determining the three-dimensional spatial trajectory of each object in the target video. When the target video covers multiple dynamic scenes, this embodiment of the disclosure will also generate questions and corresponding answers based on the three-dimensional spatial trajectory, targeting multiple motion attributes (such as distance, direction, speed, pose, etc.), and use the questions and corresponding answers as question-answer data pairs, so that the question-answer data pairs can cover not only multiple dynamic scenes, but also multiple motion attributes. Based on this, compared with existing spatial reasoning datasets that cover simple, controlled scenes and can only cover some simple questions, the spatial reasoning dataset generated by this embodiment of the disclosure based on target videos and multiple question-answer data pairs corresponding to the target videos covers multiple dynamic scenes and multiple motion attributes. By training a visual language model using this spatial reasoning dataset, the visual language model can understand and reason about changes in spatial relationships between objects in dynamic scenes.

[0168] The electronic device provided in this disclosure for executing the above-described method for constructing spatial reasoning datasets can be a terminal, referring to... Figure 12 , Figure 12 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure. The terminal includes: a camera assembly 1210, a first memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a first processor 1280, and a first power supply 12120, among other components. Those skilled in the art will understand that... Figure 12 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0169] The camera assembly 1210 can be used to capture images or videos. Optionally, the camera assembly 1210 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions.

[0170] The first memory 1220 can be used to store software programs and modules. The first processor 1280 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1220.

[0171] The input unit 1230 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232.

[0172] The display unit 1240 can be used to display input or provided information, as well as various menus of the terminal. The display unit 1240 may include a display panel 1241.

[0173] Audio circuitry 1260, speaker 1261, and microphone 1262 provide an audio interface.

[0174] The first power source 1290 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0175] The number of sensors 1250 can be one or more, and these sensors 1250 include, but are not limited to: accelerometers, gyroscopes, pressure sensors, optical sensors, etc. An accelerometer can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, an accelerometer can be used to detect the components of gravitational acceleration along the three coordinate axes. The first processor 1280 can control the display unit 1240 to display the user interface in either a horizontal or vertical view based on the gravitational acceleration signal acquired by the accelerometer. The accelerometer can also be used for collecting motion data from games or other applications.

[0176] The gyroscope sensor can detect the terminal's orientation and rotation angle. It can work in conjunction with an accelerometer to collect 3D user movements on the terminal. Based on the data collected by the gyroscope sensor, the first processor 1280 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0177] The pressure sensor can be installed on the side bezel of the terminal and / or on the lower layer of the display unit 1240. When the pressure sensor is installed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the first processor 1280 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor. When the pressure sensor is installed on the lower layer of the display unit 1240, the first processor 1280 can control the operable controls on the UI interface based on the user's pressure operation on the display unit 1240. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0178] An optical sensor is used to collect ambient light intensity. In one embodiment, the first processor 1280 can control the display brightness of the display unit 1240 based on the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1240 is increased; when the ambient light intensity is low, the display brightness of the display unit 1240 is decreased. In another embodiment, the first processor 1280 can also dynamically adjust the shooting parameters of the camera assembly 1210 based on the ambient light intensity collected by the optical sensor.

[0179] In this embodiment, the first processor 1280 included in the terminal can execute the spatial reasoning dataset construction method of the previous embodiment.

[0180] The electronic device provided in this disclosure for executing the above-described method for constructing spatial reasoning datasets can also be a server, see below. Figure 13 , Figure 13This is a partial structural block diagram of a server provided in an embodiment of the present disclosure. The server can vary significantly due to different configurations or performance. It may include one or more second processors 1310 and second memories 1330, and one or more storage media 1340 (e.g., one or more mass storage devices) for storing application programs 1343 or data 1342. The second memories 1330 and storage media 1340 may be temporary or persistent storage. The program stored in the storage media 1340 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the second processor 1310 may be configured to communicate with the storage media 1340 and execute the series of instruction operations stored in the storage media 1340 on the server.

[0181] The server may also include one or more secondary power supplies 1320, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1360, and / or one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0182] The second processor 1310 in the server can be used to execute the method for constructing the spatial inference dataset.

[0183] This disclosure also provides a computer-readable storage medium for storing a computer program for executing the spatial reasoning dataset construction method of the foregoing embodiments.

[0184] This disclosure also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method for constructing the spatial inference dataset described above.

[0185] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate to describe embodiments of this disclosure, for example, those that can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0186] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0187] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0188] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects. The above is a detailed description of the preferred embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method for constructing a spatial reasoning dataset, characterized in that, The method comprises: obtaining a plurality of target videos, the target videos being videos in which objects move; for each target video, determining a three-dimensional space trajectory of each object in the target video; generating, based on the three-dimensional space trajectory, a question asking about a plurality of motion attributes, and an answer corresponding to the question, and taking the question and the answer corresponding to the question as a question and answer data pair, wherein the motion attribute indicates an attribute of an object that changes in three-dimensional space; generating, based on the target video and a plurality of question and answer data pairs corresponding to the target video, a space reasoning data set.

2. The method of claim 1, wherein, The determination of the three-dimensional space trajectory of each object in the target video comprises: extracting, in the target video, a camera pose of a shooting camera at each video frame, a point cloud distribution of each video frame, an object mask in each video frame, and an object pose of each object in each video frame; determining, based on the camera pose, the point cloud distribution, the object mask, and the object pose, the three-dimensional space trajectory of each object in the target video.

3. The method of claim 2, wherein, The extraction of the camera pose of the shooting camera at each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame in the target video comprises: determining, based on a video description text corresponding to the target video, a category of each object in the target video; uniformly sampling the target video to obtain a video frame sequence, the video frame sequence comprising a plurality of continuous video frames; based on the category of each object in the target video, extracting, from the video frame sequence, the camera pose of the shooting camera at each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame in the target video.

4. The method of claim 3, wherein, The extraction of the camera pose of the shooting camera at each video frame, the point cloud distribution of each video frame, the object mask in each video frame, and the object pose of each object in each video frame in the target video comprises: analyzing the video frame sequence using a visual understanding model to obtain the camera pose of the shooting camera and the point cloud distribution of each video frame; analyzing the video frame sequence and the category of the object using a mask extraction model to obtain the object mask of each video frame; analyzing the video frame sequence using a pose analysis model to obtain the object pose of each object in each video frame.

5. The method of claim 2, wherein, The determination of the three-dimensional space trajectory of each object in the target video based on the camera pose, the point cloud distribution, the object mask, and the object pose comprises: based on the object mask in each video frame, screening out the point cloud of the object from the point cloud distribution of each video frame; for each object in each video frame, determining the position of the object in three-dimensional space based on the point cloud of the object; The positions of each object in each video frame in the three-dimensional space are sequentially connected according to the time sequence of the video frames, to obtain a three-dimensional space trajectory of each object in the target video.

6. The method of claim 5, wherein, The position of the object in the three-dimensional space is determined based on the point cloud of the object, including: The three-dimensional space coordinates of each point in the point cloud of the object are obtained; The average value of the three-dimensional space coordinates of each point is taken as the position of the object in the three-dimensional space.

7. The method of claim 1, wherein, The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including: The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including: The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including:

8. The method of claim 7, wherein, The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including: The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including: The preset template includes an absolute perspective template and a relative perspective template, the absolute perspective template is a question template asked for a motion attribute with a static reference system as an observation reference, and the relative perspective template is a question template asked for a motion attribute with a dynamic reference system as an observation reference; The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including:

9. The method of claim 8, wherein, When the preset template is the absolute perspective template, a static reference system to which the question is directed, a target object to which the question is directed, and a time range to which the question is directed are determined, and the absolute perspective template is filled based on the static reference system, the target object, the time range, and the motion attribute, to obtain the question text asked for the motion attribute; When the preset template is the relative perspective template, a dynamic reference system to which the question is directed, a target object to which the question is directed, and a time range to which the question is directed are determined, and the relative perspective template is filled based on the dynamic reference system, the target object, the time range, and the motion attribute, to obtain the question text asked for the motion attribute. The question asked for each motion attribute is obtained by filling a preset template based on the motion attribute, and the answer corresponding to the question is obtained by calculating the question asked for the motion attribute based on the three-dimensional space trajectory, including: The three-dimensional space trajectory of each object is converted to a reference system to which the question is directed, to obtain a converted three-dimensional space trajectory of each object; 10. The method of claim 9, wherein, A plurality of continuous video frames corresponding to the time range to which the question is directed are determined; ​ ​ For adjacent video frames, the change of motion attributes within adjacent video frames is calculated based on the converted three-dimensional space track, to obtain a problem sub-answer; The plurality of problem sub-answers are integrated to obtain an answer corresponding to the question.

11. The method of claim 1, wherein, The plurality of target videos include: Obtaining a video data set, the video data set including a plurality of videos; Filtering target videos with object motion from the video data set.

12. The method of claim 11, wherein, The filtering target videos with object motion from the video data set includes: For each video in the video data set, obtaining a video description text corresponding to the video, the video description text being a text describing an event occurring in the video; Detecting whether there is a description of object motion in the video description text, and when there is a description of object motion in the video description text, determining the video as the target video.

13. The method of claim 11, wherein, The filtering target videos with object motion from the video data set includes: For each video in the video data set, obtaining a plurality of video frames of the video; Based on a plurality of video frames, determining whether there is object motion in the video, when there is object motion in the video, determining the video as the target video.

14. The method of claim 1, wherein, The generating a spatial reasoning data set based on the target video and a plurality of question and answer data pairs corresponding to the target video includes: The first number of target videos and the question and answer data pairs corresponding to the target video are used as a training data set, and the second number of target videos and the question and answer data pairs corresponding to the target video are used as an evaluation data set, and the second number is less than the first number; The training data set and the evaluation data set are integrated to obtain the spatial reasoning data set.

15. The method of claim 1, wherein, After generating the spatial reasoning data set based on the target video and a plurality of question and answer data pairs corresponding to the target video, it further includes: Training a visual language model based on the spatial reasoning data set; The visual language model reasons a question in the following way: Receiving a spatial reasoning question and a spatial reasoning video; Converting the spatial reasoning question into a text feature vector and converting the spatial reasoning video into a three-dimensional geometric feature vector; Performing semantic compression on the text feature vector to obtain a compressed text feature vector; Extracting a target three-dimensional geometric feature vector related to the compressed text feature vector from the three-dimensional geometric feature vector; Based on the text feature vector and the target three-dimensional geometric feature vector, performing fusion reasoning to obtain an answer corresponding to the spatial reasoning question.

16. The method of claim 15, wherein, The converting the spatial reasoning video into a three-dimensional geometric feature vector includes: Converting the spatial reasoning video into a visual feature vector; Based on the visual feature vector, extracting a three-dimensional geometric feature vector in the spatial reasoning video; The based on the text feature vector and the target three-dimensional geometric feature vector, performing fusion reasoning to obtain an answer corresponding to the spatial reasoning question includes: Fusion reasoning is performed based on the text feature vector, the visual feature vector and the target three-dimensional geometric feature vector to obtain a solution corresponding to the spatial reasoning problem.

17. A spatially-reasoning dataset construction apparatus, comprising: The method comprises the following steps: An acquisition module is configured to acquire a plurality of target videos, the target videos being videos in which objects move; A determination module is configured to determine, for each target video, a three-dimensional spatial trajectory of each object in the target video; A first generation module is configured to generate, based on the three-dimensional spatial trajectory, a question for a plurality of motion attributes, an answer corresponding to the question, and a question and answer data pair comprising the question and the answer corresponding to the question, wherein the motion attribute indicates an attribute of the object that changes in three-dimensional space; A second generation module is configured to generate, based on the target video and a plurality of question and answer data pairs corresponding to the target video, a spatial reasoning data set.

18. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the method for constructing the spatial reasoning data set according to any one of claims 1 to 16.

19. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method for constructing the spatial reasoning data set according to any one of claims 1 to 16.

20. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method for constructing the spatial reasoning data set according to any one of claims 1 to 16.

Images