Scene investigation record generation method and device, equipment and storage medium
By constructing a three-dimensional scene model and generating detailed records, the problems of low efficiency in writing current survey records and incomplete information are solved, efficient integration and intuitive presentation of on-site survey information are achieved, and case handling efficiency is improved.
Patent Information
- Application Number
- CN202410821695.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2024-11-01
AI Technical Summary
The existing survey records are inefficient and incomplete, making it difficult to preserve the case site for a long time, and the scene is not intuitive to reproduce, which can easily lead to errors in the transcript or missing information.
By obtaining on-site survey images, target recognition and three-dimensional scene model construction, three-dimensional scene information is generated, and inputting it into text generation model, generating detailed on-site survey records, combining VR technology and deep learning algorithms to realize the reconstruction of three-dimensional scenes and Information integration.
It improves the information integrity and production efficiency of the current survey records, ensures the accuracy and intuitiveness of the records, and enhances case handling efficiency.
Smart Images

Figure CN118887352A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to artificial intelligence technology and virtual reality technology, and in particular to a method, device, equipment and storage medium for generating on-site investigation records. Background Art
[0002] The on-site investigation record is an evidentiary document recorded by the case-handling personnel on the results of the investigation and inspection of the case scene. At present, the on-site investigation record is manually compiled by the case-handling personnel based on the results of the on-site investigation or based on the images, videos and other materials taken at the scene.
[0003] However, for some complex cases, investigators need to go to the scene of the case several times to conduct repeated investigations and case deductions, but the scene of the case is difficult to preserve for a long time. If the scene is changed or damaged, the content of the record will be wrong and incomplete. It is difficult to reproduce the scene of the case by relying on images and videos. The scene restoration is not intuitive enough, the efficiency of record writing is low, and there may be information missing. Summary of the invention
[0004] The embodiments of the present disclosure provide a method, apparatus, device and storage medium for generating a field survey record, so as to improve the integrity of field survey record information and the efficiency of record production.
[0005] One aspect of the embodiments of the present disclosure provides a method for generating a field survey record, comprising:
[0006] Acquiring a site survey image, wherein the site survey image refers to an image obtained by photographing the site environment at at least one point of the survey site;
[0007] Performing target recognition on the on-site survey image to determine on-site objects in the survey scene;
[0008] Based on the on-site survey images of each viewing angle corresponding to at least one on-site object, a three-dimensional scene model of the survey site is generated and three-dimensional scene information is obtained, wherein the three-dimensional scene information includes object category, size information, and orientation information of a virtual object in the three-dimensional scene model;
[0009] The three-dimensional scene information is input into a text generation model to generate a site survey record, wherein the site survey record contains a description text of the three-dimensional scene model.
[0010] Optionally, the on-site objects include movable items and building structures;
[0011] The generating of a three-dimensional scene model of the survey scene and obtaining three-dimensional scene information based on the site survey images of each viewing angle corresponding to at least one site object comprises:
[0012] Generate a three-dimensional space point cloud of at least one movable object based on the on-site survey images of various viewing angles corresponding to the movable object;
[0013] Generating a virtual reality (VR) layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure;
[0014] The three-dimensional scene model is generated based on the three-dimensional space point cloud and the VR layout model, and the three-dimensional scene model is parsed to obtain the three-dimensional scene information.
[0015] Optionally, generating a three-dimensional space point cloud of the movable object based on the on-site survey images of each viewing angle corresponding to at least one movable object includes:
[0016] Acquire close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are obtained by intercepting an area where the movable object is located in the on-site investigation image;
[0017] Using VR multi-view pose stitching technology to stitch the close-up image of at least one movable object to obtain a VR multi-view stitching result;
[0018] Generate an initial space point cloud based on a VR dimension conversion algorithm, a monocular depth estimation algorithm and the VR multi-view stitching result;
[0019] The initial space point cloud is clustered and filtered to generate the three-dimensional space point cloud.
[0020] Optionally, generating a VR layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure includes:
[0021] Based on the on-site survey images of various viewing angles corresponding to the at least one building structure, a three-dimensional model of the at least one building structure is generated using a VR dimension conversion algorithm;
[0022] The three-dimensional model of the building structure is adjusted based on the floor plan of the survey site to generate the VR layout model, wherein the floor plan includes the outline and relative position relationship of the building structure in the survey site.
[0023] Optionally, generating the three-dimensional scene model based on the three-dimensional space point cloud and the VR layout model, and parsing the three-dimensional scene model to obtain the three-dimensional scene information includes:
[0024] The three-dimensional space point cloud and the VR layout model are integrated to optimize the placement of the movable objects to obtain the three-dimensional scene model;
[0025] Parsing the three-dimensional scene model;
[0026] The three-dimensional scene information is generated based on the three-dimensional partition information of each model partition obtained by analysis, wherein the three-dimensional partition information includes object category, size information and orientation information of virtual objects in the model partition, and the virtual objects include virtual movable objects and virtual building structures.
[0027] Optionally, the parsing the three-dimensional scene model and generating the three-dimensional scene information based on the three-dimensional partition information of each model partition includes:
[0028] Parsing the three-dimensional scene model, and generating a partition connectivity graph based on connectivity relationships between partitions of the model;
[0029] Sorting the model partitions based on the partition connectivity graph to determine the order of information description;
[0030] The three-dimensional partition information of each model partition is combined in the order of the information description to generate the three-dimensional scene information.
[0031] Optionally, the method further includes:
[0032] Generate a plane display diagram corresponding to the three-dimensional scene model, wherein the plane display diagram is used to display the overall layout of the three-dimensional scene model;
[0033] In response to the plan view display operation, the plan view is displayed through the transcript display interface.
[0034] Another aspect of the present disclosure provides a device for generating a field survey record, including:
[0035] An acquisition module is used to acquire a site survey image, wherein the site survey image refers to an image obtained by photographing the site environment at at least one point of the survey site;
[0036] A recognition module, used to perform target recognition on the on-site survey image and determine on-site objects in the survey site;
[0037] An information generation module, configured to generate a three-dimensional scene model of the survey site and obtain three-dimensional scene information based on the site survey images of each viewing angle corresponding to at least one site object, wherein the three-dimensional scene information includes object category, size information, and orientation information of virtual objects in the three-dimensional scene model;
[0038] The record generation module is used to input the three-dimensional scene information into the text generation model to generate a field investigation record, wherein the field investigation record contains a description text of the three-dimensional scene model.
[0039] Optionally, the on-site objects include movable items and building structures;
[0040] The information generation module is further used for:
[0041] Generate a three-dimensional space point cloud of at least one movable object based on the on-site survey images of various viewing angles corresponding to the movable object;
[0042] Generate a VR layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure;
[0043] The three-dimensional scene model is generated based on the three-dimensional space point cloud and the VR layout model, and the three-dimensional scene model is parsed to obtain the three-dimensional scene information.
[0044] Optionally, the information generating module is further used to:
[0045] Acquire close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are obtained by intercepting an area where the movable object is located in the on-site investigation image;
[0046] Acquire close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are obtained by intercepting an area where the movable object is located in the on-site investigation image;
[0047] Using VR multi-view pose stitching technology to stitch the close-up image of at least one movable object to obtain a VR multi-view stitching result;
[0048] Generate an initial space point cloud based on a VR dimension conversion algorithm, a monocular depth estimation algorithm and the VR multi-view stitching result;
[0049] The initial space point cloud is clustered and filtered to generate the three-dimensional space point cloud.
[0050] Optionally, the information generating module is further used to:
[0051] Based on the on-site survey images of various viewing angles corresponding to the at least one building structure, a three-dimensional model of the at least one building structure is generated using a VR dimension conversion algorithm;
[0052] The three-dimensional model of the building structure is adjusted based on the floor plan of the survey site to generate the VR layout model, wherein the floor plan includes the outline and relative position relationship of the building structure in the survey site.
[0053] Another aspect of the disclosed embodiment provides a system for generating a field survey record, the system comprising an image acquisition device and a record generating device;
[0054] The image acquisition device is used to acquire a scene investigation image and transmit the scene investigation image to the record generation device, wherein the scene investigation image refers to an image obtained by photographing the scene environment at at least one point of the investigation scene;
[0055] The record generation device is used to obtain the on-site investigation image; perform target recognition on the on-site investigation image to determine the on-site objects in the investigation site; generate a three-dimensional scene model of the investigation site and obtain three-dimensional scene information based on the on-site investigation images of various perspectives corresponding to at least one on-site object, wherein the three-dimensional scene information includes the object category, size information and orientation information of the virtual object in the three-dimensional scene model; input the three-dimensional scene information into a text generation model to generate an on-site investigation record, wherein the on-site investigation record includes a description text of the three-dimensional scene model.
[0056] Another aspect of the present disclosure provides an electronic device, including:
[0057] Memory for storing computer programs;
[0058] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method described in the above aspects is implemented.
[0059] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the above aspects is implemented.
[0060] Another aspect of the embodiments of the present disclosure provides a computer program, including computer program instructions, which implement the methods described in the above aspects when executed by a processor.
[0061] Based on the disclosed embodiment, a solution for automatically generating on-site investigation records based on three-dimensional scene reconstruction is provided. By performing target recognition on the on-site investigation images obtained by on-site shooting, a three-dimensional scene model of the investigation site is constructed based on the identified on-site objects, thereby obtaining three-dimensional scene information. The three-dimensional scene information is input into the text generation model to generate a coherent on-site investigation record. With the help of three-dimensional real scenes, users can quickly generate on-site investigation records, which can ensure the integrity of the record information and improve the efficiency of making and handling on-site investigation records.
[0062] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0064] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:
[0065] Figure 1 A flowchart of an embodiment of a method for generating a field survey record disclosed in the present invention;
[0066] Figure 2 A flowchart of another embodiment of the method for generating on-site inspection records disclosed herein;
[0067] Figure 3 A schematic diagram of the order of integrating three-dimensional scene information in an embodiment of the present disclosure;
[0068] Figure 4 A schematic diagram of a transcript display interface in an embodiment of the present disclosure;
[0069] Figure 5 A schematic diagram of the structure of an embodiment of a device for generating on-site inspection records disclosed in the present invention;
[0070] Figure 6 The figure is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION
[0071] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.
[0072] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0073] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0074] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0075] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.
[0076] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0077] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0078] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0079] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0080] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0081] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0082] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.
[0083] Please refer to Figure 1 , which shows a flow chart of a method for generating a field survey record provided by an exemplary embodiment of the present disclosure. The generation of the field survey record of the embodiment of the present disclosure can be implemented by an electronic device equipped with a target detection model and a text generation model. The embodiment includes the following steps:
[0084] Step 101 : Acquire a site survey image, where the site survey image refers to an image obtained by photographing the site environment at at least one point of the survey site.
[0085] Optionally, the electronic device in the embodiment of the present disclosure may be a background server, and the user may collect on-site survey images through a mobile device and upload them to the electronic device; or, the electronic device may be a mobile device of the user, and the mobile device may directly collect or obtain previously collected on-site survey images from local memory.
[0086] In a possible implementation, a point refers to a location at which a field survey image is taken at the survey site. In order to restore the survey site as much as possible, the user may collect field survey images at multiple points, for example, taking at least one field survey image at intervals of 2 meters in at least two directions.
[0087] Step 102: Perform target recognition on the on-site survey image to determine on-site objects in the survey site.
[0088] In a possible implementation, the site survey image is input into a target detection model based on a deep learning algorithm to perform target recognition and determine the site objects contained in the image, such as walls, windows, tables, etc. The image area corresponding to the site object in the site survey image is determined and target recognition is performed on the area to obtain the category of the object.
[0089] Step 103: Based on the on-site survey images of various viewing angles corresponding to at least one on-site object, a three-dimensional scene model of the survey site is generated and three-dimensional scene information is obtained.
[0090] The three-dimensional scene information includes object category, size information and orientation information of virtual objects in the three-dimensional scene model.
[0091] Optionally, after identifying the on-site object, the on-site survey images containing the same on-site object are determined. For at least one on-site object, the image areas where the object is located in all the corresponding on-site survey images are spliced, and a VR algorithm is used to transform from two-dimensional to three-dimensional to obtain a three-dimensional stereo model of the object, and then a three-dimensional scene model of the entire survey site is constructed. By parsing the three-dimensional scene model, three-dimensional scene information is obtained. The three-dimensional scene information includes the object category, size information, and orientation information of the virtual object in the three-dimensional scene model, such as the orientation and connectivity relationship of each room in the three-dimensional scene model, and the orientation and position of each object in at least one room.
[0092] Step 104 , input the three-dimensional scene information into a text generation model to generate a site survey record, wherein the site survey record includes a description text of the three-dimensional scene model.
[0093] The three-dimensional scene information is an information fragment containing the object category, size information and orientation information of the virtual objects in the three-dimensional scene model. By inputting the three-dimensional scene information into the text generation model, a coherent description text can be generated, and other information such as the record number and trace evidence can be combined to obtain a human-readable on-site investigation record. That is, the description text contains the three-dimensional scene information such as the object category, size information and orientation information of each on-site object in the investigation scene, and can also contain other supplementary information such as physical evidence traces uploaded by users, investigation conditions, and investigation time.
[0094] Optionally, after the on-site investigation record is generated, the content of the on-site investigation record can be displayed through the application interface, and a file can be generated for the user to download and save.
[0095] In the disclosed embodiment, by performing target recognition on the on-site survey image obtained by on-site shooting, a three-dimensional scene model of the survey site is constructed based on the identified on-site objects, thereby obtaining three-dimensional scene information, and the three-dimensional scene information is input into the text generation model to generate a coherent on-site survey record. With the help of three-dimensional real scenes, users can quickly generate on-site survey records, which can ensure the integrity of the record information and improve the efficiency of making on-site survey records and handling cases.
[0096] In a possible implementation, the objects on site mainly include movable objects (objects placed in space relative to fixed building structures, such as tables, cabinets, televisions, cups, etc., and objects fixed to building structures by welding, pasting, etc.) and relatively fixed building structures (such as walls, doors, windows, etc.). By performing 3D reconstruction on the above two types of objects respectively, the 3D scene information of the complete survey site can be obtained. Figure 2 A flowchart of a method for generating a field survey record provided by another exemplary embodiment of the present application is shown. The generation of the field survey record of the embodiment of the present disclosure can be implemented by an electronic device equipped with a target detection model and a text generation model. Figure 1 Based on the illustrated embodiment, the above step 103 may include the following steps:
[0097] Step 103a: Generate a three-dimensional space point cloud of the movable object based on the on-site survey images of various viewing angles corresponding to at least one movable object.
[0098] A three-dimensional point cloud is a data structure used to represent an object or scene in a three-dimensional space. It is a collection of multiple three-dimensional coordinate points. Schematically, by obtaining the features of key areas of objects in the image (such as edges, corners, etc.), the VR algorithm is used to calculate the corresponding positions of each feature point in the three-dimensional space to generate a three-dimensional point cloud. In order to improve the accuracy and rationality of the three-dimensional point cloud, VR multi-view stitching can be performed by combining on-site survey images of multiple perspectives corresponding to the same object. The specific step 103a includes the following steps:
[0099] Step 1: acquiring close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are acquired by intercepting the movable object region in the on-site investigation image.
[0100] In a possible implementation, all on-site investigation images containing the same movable object are first determined, and then a close-up image of the movable object is captured from the on-site investigation images based on the image region where the movable object is located.
[0101] Step 2: Use VR multi-view pose stitching technology to stitch at least one close-up image of a movable object to obtain a VR multi-view stitching result.
[0102] Different on-site survey images are taken at different points or directions, and therefore correspond to different perspectives. VR multi-perspective pose stitching technology is used to stitch at least one close-up image of a movable object to obtain a VR multi-perspective stitching result, which can assist in generating a more complete and accurate three-dimensional point cloud.
[0103] Step 3: Generate the initial spatial point cloud based on the VR dimension conversion algorithm, monocular depth estimation algorithm and VR multi-view stitching results;
[0104] Step 4: Cluster and filter the initial spatial point cloud, aggregate the point clouds corresponding to the movable objects in the initial spatial point cloud, filter out the noise points, and generate a three-dimensional spatial point cloud.
[0105] The VR dimension conversion algorithm is an algorithm used to convert a two-dimensional image into a three-dimensional space to obtain a three-dimensional point cloud, while the monocular depth estimation algorithm refers to a technology that uses a two-dimensional image captured by a single camera to estimate the distance and depth of objects in the image.
[0106] By combining the VR dimension conversion algorithm, the monocular depth estimation algorithm and the VR multi-view stitching results, an initial spatial point cloud containing the corresponding point clouds of each movable object is generated. Furthermore, the three-dimensional spatial point cloud processing algorithm is used to cluster and filter the initial spatial point cloud to obtain a clearer and more accurate three-dimensional spatial point cloud.
[0107] Step 103b: Generate a VR layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure.
[0108] In order to restore the building structure on site and obtain accurate spatial position relationships, a VR layout model of the survey site can also be generated using a VR algorithm based on the on-site survey images of various perspectives corresponding to at least one building structure.
[0109] In a possible implementation, the three-dimensional model of the building structure can be modified using the floor plan of the survey site to obtain an accurate VR layout model. Step 103b specifically includes the following steps:
[0110] Step 5: Based on the on-site survey images of various viewing angles corresponding to at least one building structure, a three-dimensional model of at least one building structure is generated using a VR dimension conversion algorithm.
[0111] Optionally, a VR 2D to 3D algorithm is used to determine the positions of key building structures such as walls, doors, and windows in 3D space and generate a corresponding 3D model.
[0112] Step six: adjust the three-dimensional model of the building structure based on the floor plan of the survey site to generate a VR layout model. The floor plan includes the outline and relative position relationship of the building structure in the survey site.
[0113] Based on the precise positions and outlines of the building structures such as the partition relationship, wall lines, doors and windows in the floor plan at the site of the survey, the 3D model of the building structure is adjusted to generate a VR layout model. The partition refers to the separate spaces such as the living room, bedroom, and bathroom separated by walls and door structures.
[0114] Optionally, if the on-site objects identified from the on-site investigation image do not include building structures (for example, the crime occurred outdoors), there is no need to execute step 103b, and a three-dimensional scene model is directly constructed based on the three-dimensional point cloud of the movable objects and the three-dimensional scene information is obtained to generate a scene investigation record.
[0115] Step 103c, generating a three-dimensional scene model based on the three-dimensional space point cloud and the VR layout model, and parsing the three-dimensional scene model to obtain three-dimensional scene information.
[0116] Based on the 3D point cloud and VR layout model, a complete 3D scene model can be constructed, which includes virtual movable objects and virtual building structures. By parsing the 3D scene model and obtaining information such as the orientation, position, and size of the objects, key structured information for writing on-site survey notes can be obtained.
[0117] In a possible implementation, the three-dimensional scene model may be disassembled to generate organized three-dimensional scene information according to a preset compartment arrangement order. Step 103c specifically includes the following steps:
[0118] Step seven: fuse the three-dimensional space point cloud and the VR layout model, optimize the placement of movable objects, and obtain a three-dimensional scene model.
[0119] The 3D model generated directly based on the fusion of the 3D space point cloud and the VR layout model may have unreasonable structures, such as intersections between different objects, or objects hanging in the air, etc. Therefore, after the model is fused, the placement of movable objects can be optimized to obtain a 3D scene model with a reasonable structure.
[0120] Step eight, analyzing the three-dimensional scene model.
[0121] Step nine, based on the three-dimensional partition information of each model partition obtained by analysis, generate three-dimensional scene information, the three-dimensional partition information includes the object category, size information and orientation information of the virtual objects in the model partition, and the virtual objects include virtual movable objects and virtual building structures.
[0122] The on-site survey record is usually described in a certain spatial order and description rules, such as starting from the compartment where the entrance door is located, and describing the structure, orientation, and placement of items in each compartment in a clockwise order. Therefore, the 3D scene model can be analyzed and disassembled into a space composed of compartments, and then the 3D scene information can be generated based on the 3D compartment information of each model compartment.
[0123] In a possible implementation, step seven may include the following steps:
[0124] Step a: parse the three-dimensional scene model and generate a partition connectivity graph based on the connectivity relationship between the model partitions.
[0125] The connectivity relationship between model compartments is used to indicate whether there is a passage for people to travel back and forth between them. For example, if there is a door in the wall between the living room and the bedroom, the living room and the bedroom are connected through the door. The compartment connectivity graph contains the connectivity relationship between model compartments.
[0126] Step b: sort the model partitions based on the partition connectivity graph to determine the order of information description.
[0127] Optionally, the model compartments may be sorted according to a preset spatial arrangement order and a compartment connectivity diagram to determine the order of information description.
[0128] Indicatively, the model compartments are sorted in a clockwise order starting from the compartment where the entrance door is located. For example, if the compartment where the entrance door is located is the living room, the living room is determined to be the first model compartment; the house as a whole faces south, and the east wall of the living room is detected in a clockwise order to see if it is connected to another compartment. If it is detected that the east wall of the living room is connected to the kitchen through a one-way door, the kitchen is determined to be the second compartment; continue to detect whether the other walls of the kitchen are connected to other compartments in a clockwise direction. If all the walls of the kitchen are not connected to other compartments, return to the living room to confirm whether there is a compartment connected to the second wall in the clockwise direction (i.e., the south wall).
[0129] Step c: combining the three-dimensional partition information of each model partition in the order of information description to generate three-dimensional scene information.
[0130] The three-dimensional scene information is obtained by respectively obtaining the three-dimensional partition information of each model partition and then combining them in the order of information description. Figure 3 The schematic diagram of the integration process of three-dimensional scene information and physical evidence traces is shown. The process in the figure first determines the information description order from compartment 1 to compartment n based on the overall structure and orientation of the investigation site. For at least one compartment, the direction, type and material of the door, the wall where the door is located, and the physical evidence traces at the door position are determined starting from the door corresponding to the compartment. Then the furniture against the wall where the door is located, the physical evidence traces against the wall where the door is located, the furniture against the wall of the first wall in the clockwise direction, the physical evidence traces against the wall of the first wall in the clockwise direction, the door type of the first wall in the clockwise direction, and the connected compartments. After the furniture, physical evidence traces corresponding to the four walls and the connected compartments are connected in series, the furniture and physical evidence traces at the center of the compartment are determined, and so on, until the three-dimensional compartment information of the last compartment is connected in series, and the complete three-dimensional scene information is obtained. Among them, the physical evidence traces and the locations of the physical evidence traces can be uploaded by the user.
[0131] In the disclosed embodiment, a three-dimensional point cloud of an object is generated by combining a VR dimension conversion algorithm, a monocular depth estimation algorithm, and a VR multi-view stitching result. At the same time, a precise VR layout model is constructed by combining the VR dimension conversion algorithm with the floor plan. This can accurately restore the scene of the survey site and further improve the completeness and accuracy of the survey record.
[0132] In a possible implementation, considering that images can reflect scene information more vividly than text, a display image corresponding to the three-dimensional scene model can also be generated while generating the on-site inspection record. Figure 1 Based on the illustrated embodiment, the method of the embodiment of the present disclosure may further include the following steps:
[0133] Step ten: Generate a plane display diagram corresponding to the three-dimensional scene model, where the plane display diagram is used to display the overall layout of the three-dimensional scene model.
[0134] Step eleven, in response to the plan display operation, display the plan display map through the record display interface.
[0135] Optionally, the three-dimensional scene model may be globally photographed (eg, photographed from a bird's-eye view) by a virtual camera to obtain a planar display image.
[0136] Indicative, Figure 4 A schematic diagram of a record display interface is shown. The left side of the interface in the figure is a text display area, which is used to display the text content of the on-site survey record. The user can download the file containing the text content of the on-site survey record by triggering the "Download" control below. The right side is other information display, such as the "floor plan" shown below, which displays the layout and object distribution of the entire survey scene in the form of a two-dimensional floor plan.
[0137] The plan display diagram can be used to obtain information about the survey scene more intuitively, and the plan display diagram covers the entire survey scene, making it convenient for relevant personnel to quickly obtain global information and improving the richness of the content of the on-site survey record.
[0138] Please refer to Figure 5 , which shows a structural block diagram of a device for generating on-site inspection records provided by an exemplary embodiment of the present disclosure. The device includes:
[0139] The acquisition module 501 is used to acquire a site survey image, where the site survey image refers to an image obtained by photographing the site environment at at least one point of the survey site;
[0140] The recognition module 502 is used to perform target recognition on the scene survey image to determine the scene objects in the survey scene;
[0141] An information generation module 503 is used to generate a three-dimensional scene model of the survey site and obtain three-dimensional scene information based on the site survey images of each viewing angle corresponding to at least one site object, wherein the three-dimensional scene information includes object category, size information and orientation information of the virtual object in the three-dimensional scene model;
[0142] The record generation module 504 is used to input the three-dimensional scene information into the text generation model to generate a site investigation record, wherein the site investigation record contains a description text of the three-dimensional scene model.
[0143] Optionally, the on-site objects include movable items and building structures;
[0144] The information generating module 503 is further used for:
[0145] Generate a three-dimensional space point cloud of at least one movable object based on the on-site survey images of various viewing angles corresponding to the movable object;
[0146] Generate a VR layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure;
[0147] The three-dimensional scene model is generated based on the three-dimensional space point cloud and the VR layout model, and the three-dimensional scene model is parsed to obtain the three-dimensional scene information.
[0148] Optionally, the information generating module 503 is further used to:
[0149] Acquire close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are obtained by intercepting an area where the movable object is located in the on-site investigation image;
[0150] Acquire close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are obtained by intercepting an area where the movable object is located in the on-site investigation image;
[0151] Using VR multi-view pose stitching technology to stitch the close-up image of at least one movable object to obtain a VR multi-view stitching result;
[0152] Generate an initial space point cloud based on a VR dimension conversion algorithm, a monocular depth estimation algorithm and the VR multi-view stitching result;
[0153] The initial space point cloud is clustered and filtered to generate the three-dimensional space point cloud.
[0154] Optionally, the information generating module 503 is further used to:
[0155] Based on the on-site survey images of various viewing angles corresponding to the at least one building structure, a three-dimensional model of the at least one building structure is generated using a VR dimension conversion algorithm;
[0156] The three-dimensional model of the building structure is adjusted based on the floor plan of the survey site to generate the VR layout model, wherein the floor plan includes the outline and relative position relationship of the building structure in the survey site.
[0157] Optionally, the information generating module 503 is further used to:
[0158] The three-dimensional space point cloud and the VR layout model are integrated to optimize the placement of the movable objects to obtain the three-dimensional scene model;
[0159] Parsing the three-dimensional scene model;
[0160] The three-dimensional scene information is generated based on the three-dimensional partition information of each model partition obtained by analysis, wherein the three-dimensional partition information includes object category, size information and orientation information of virtual objects in the model partition, and the virtual objects include virtual movable objects and virtual building structures.
[0161] Optionally, the information generating module 503 is further used to:
[0162] Parsing the three-dimensional scene model, and generating a partition connectivity graph based on connectivity relationships between partitions of the model;
[0163] Sorting the model partitions based on the partition connectivity graph to determine the order of information description;
[0164] The three-dimensional partition information of each model partition is combined in the order of the information description to generate the three-dimensional scene information.
[0165] Optionally, the device further comprises:
[0166] A plan view generation module, used to generate a plan view corresponding to the three-dimensional scene model, wherein the plan view is used to display the overall layout of the three-dimensional scene model;
[0167] The display module is used to display the plan display diagram through the record display interface in response to the plan display operation.
[0168] In addition, the embodiment of the present disclosure also provides a system for generating a field survey record, the system comprising an image acquisition device and a record generating device;
[0169] The image acquisition device is used to acquire a scene investigation image and transmit the scene investigation image to the record generation device, wherein the scene investigation image refers to an image obtained by photographing the scene environment at at least one point of the investigation scene;
[0170] The record generation device is used to obtain the on-site investigation image; perform target recognition on the on-site investigation image to determine the on-site objects in the investigation site; generate a three-dimensional scene model of the investigation site and obtain three-dimensional scene information based on the on-site investigation images of various perspectives corresponding to at least one on-site object, wherein the three-dimensional scene information includes the object category, size information and orientation information of the virtual object in the three-dimensional scene model; input the three-dimensional scene information into a text generation model to generate an on-site investigation record, wherein the on-site investigation record includes a description text of the three-dimensional scene model.
[0171] Each embodiment in this specification is described in a progressive manner, and at least one embodiment focuses on the differences from other embodiments. The same, similar or corresponding parts between the embodiments can be referred to each other. Since the system, method, device, and equipment embodiments are basically corresponding, the relevant parts can be referred to the description of the corresponding parts. The systems, methods, devices, and equipment of the embodiments of the present disclosure also correspond to each other in terms of specific implementation and beneficial technical effects. The relevant contents can be referred to each other and will not be repeated.
[0172] In addition, an embodiment of the present disclosure further provides an electronic device, including:
[0173] Memory for storing computer programs;
[0174] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method for generating the on-site survey record described in any of the above embodiments of the present disclosure is implemented.
[0175] Figure 6 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 6 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.
[0176] like Figure 6 As shown, the electronic device includes one or more processors and memory.
[0177] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0178] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the method for generating the on-site inspection record of each embodiment of the present disclosure described above and / or other desired functions.
[0179] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0180] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.
[0181] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0182] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.
[0183] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for generating on-site survey records according to various embodiments of the present disclosure described in the above part of this specification.
[0184] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0185] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for generating on-site survey records according to various embodiments of the present disclosure described in the above part of this specification.
[0186] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0187] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.
[0188] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0189] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0190] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0191] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0192] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0193] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0194] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for generating a field survey record, characterized in that: include: Acquiring a site survey image, wherein the site survey image refers to an image obtained by photographing the site environment at at least one point of the survey site; Performing target recognition on the on-site survey image to determine on-site objects in the survey scene; Based on the on-site survey images of each viewing angle corresponding to at least one on-site object, a three-dimensional scene model of the survey site is generated and three-dimensional scene information is obtained, wherein the three-dimensional scene information includes object category, size information, and orientation information of a virtual object in the three-dimensional scene model; The three-dimensional scene information is input into a text generation model to generate a site survey record, wherein the site survey record contains a description text of the three-dimensional scene model.
2. The method according to claim 1, characterized in that The objects on site include movable objects and building structures; The generating of a three-dimensional scene model of the survey scene and obtaining three-dimensional scene information based on the site survey images of each viewing angle corresponding to at least one site object comprises: Generate a three-dimensional space point cloud of at least one movable object based on the on-site survey images of various viewing angles corresponding to the movable object; Generate a virtual reality (VR) layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure; The three-dimensional scene model is generated based on the three-dimensional space point cloud and the VR layout model, and the three-dimensional scene model is parsed to obtain the three-dimensional scene information.
3. The method according to claim 2, characterized in that The generating of a three-dimensional space point cloud of the movable object based on the on-site survey images of each viewing angle corresponding to at least one movable object comprises: Acquire close-up images of various viewing angles corresponding to at least one movable object based on the on-site investigation image, wherein the close-up images are obtained by intercepting an area where the movable object is located in the on-site investigation image; Using VR multi-view pose stitching technology to stitch the close-up image of at least one movable object to obtain a VR multi-view stitching result; Generate an initial space point cloud based on a VR dimension conversion algorithm, a monocular depth estimation algorithm and the VR multi-view stitching result; The initial space point cloud is clustered and filtered to generate the three-dimensional space point cloud.
4. The method according to claim 2, characterized in that: The step of generating a VR layout model of the survey site based on the site survey images of various viewing angles corresponding to at least one building structure comprises: Based on the on-site survey images of various viewing angles corresponding to the at least one building structure, a three-dimensional model of the at least one building structure is generated using a VR dimension conversion algorithm; The three-dimensional model of the building structure is adjusted based on the floor plan of the survey site to generate the VR layout model, wherein the floor plan includes the outline and relative position relationship of the building structure in the survey site.
5. The method according to any one of claims 2 to 4, characterized in that: The generating the three-dimensional scene model based on the three-dimensional space point cloud and the VR layout model, and parsing the three-dimensional scene model to obtain the three-dimensional scene information includes: The three-dimensional space point cloud and the VR layout model are integrated to optimize the placement of the movable object to obtain the three-dimensional scene model; Parsing the three-dimensional scene model; The three-dimensional scene information is generated based on the three-dimensional partition information of each model partition obtained by analysis, wherein the three-dimensional partition information includes object category, size information and orientation information of virtual objects in the model partition, and the virtual objects include virtual movable objects and virtual building structures.
6. The method according to claim 5, characterized in that The step of parsing the three-dimensional scene model and generating the three-dimensional scene information based on the three-dimensional partition information of each model partition includes: Parsing the three-dimensional scene model, and generating a partition connectivity graph based on connectivity relationships between partitions of the model; Sorting the model partitions based on the partition connectivity graph to determine the order of information description; The three-dimensional partition information of each model partition is combined in the order of the information description to generate the three-dimensional scene information.
7. The method according to any one of claims 1 to 4, characterized in that: Also includes: Generate a plane display diagram corresponding to the three-dimensional scene model, wherein the plane display diagram is used to display the overall layout of the three-dimensional scene model; In response to the plan view display operation, the plan view is displayed through the transcript display interface.
8. A device for generating on-site survey records, characterized in that: include: An acquisition module is used to acquire a site survey image, wherein the site survey image refers to an image obtained by photographing the site environment at at least one point of the survey site; A recognition module, used to perform target recognition on the on-site survey image and determine on-site objects in the survey scene; An information generation module, configured to generate a three-dimensional scene model of the survey site and obtain three-dimensional scene information based on the site survey images of each viewing angle corresponding to at least one site object, wherein the three-dimensional scene information includes object category, size information, and orientation information of virtual objects in the three-dimensional scene model; The record generation module is used to input the three-dimensional scene information into the text generation model to generate a field investigation record, wherein the field investigation record contains a description text of the three-dimensional scene model.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.
