A multi-modal fusion spatial intelligent scene instance segmentation method and system

Abstract

The application discloses a multi-modal fusion spatial intelligent scene instance segmentation method and system, relates to the field of spatial intelligence, solves the problem of poor segmentation accuracy of scene instance segmentation technology, and comprises the following steps: S1, key word interception audio collection is carried out on a target space scene, effective interception audio in the key word interception audio is extracted through audio index analysis, and the effective interception audio is collected for a to-be-segmented instance to obtain segmented instance collection data; S2, spatial feature extraction is carried out on the to-be-segmented instance, consistency analysis is carried out on the extraction result and scene objects in the target space scene, an instance segmentation model is obtained according to an analysis result, and scene instance segmentation data is obtained; and S3, position rationality verification is carried out on a position of the instance segmentation model in the target space scene, and instance position early warning is carried out according to a verification result, so that the scene instance segmentation accuracy and segmentation efficiency can be improved.

Description

Technical Field

[0001] This invention belongs to the field of spatial intelligence and relates to multimodal fusion technology, specifically a multimodal fusion spatial intelligence scene instance segmentation method and system. Background Technology

[0002] Existing spatial intelligent scene instance segmentation technologies have the following drawbacks when performing scene instance segmentation: 1. Existing spatial intelligent scene instance segmentation technologies are mostly passive segmentation, generally lacking a professional analysis process for external interaction commands. They cannot perform intent parsing, keyword extraction, and segmentation requirement mapping for commands in the form of audio, text, etc., nor have they established accurate association rules between commands and scene instances. It is difficult to limit the segmentation target, spatial range, and accuracy requirements according to the commands as needed. They can only perform global or fixed category segmentation, which is out of touch with the interactive and customized segmentation requirements of spatial intelligence, resulting in insufficient targeting and flexibility of segmentation. 2. Existing technologies do not perform well in segmentation accuracy in complex spatial intelligent scenarios. In 3D instance segmentation, the degree of restoration of instance geometric contours is low, and the completeness and accuracy of feature point extraction and mask segmentation are insufficient. In some scenarios, there are also instances with bias in instance category determination, which makes it difficult to meet the needs of practical applications.

[0003] To address this, we propose a multimodal fusion-based spatial intelligent scene instance segmentation method and system. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multimodal fusion spatial intelligent scene instance segmentation method and system, aiming to improve instance segmentation accuracy.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal fusion spatial intelligent scene instance segmentation method, the specific steps of which are as follows: Step S1: Collect keyword-trimmed audio for the target space scene, extract the effective trimmed audio from the keyword-trimmed audio through audio index analysis, and collect segmentation instances from the effective trimmed audio to obtain segmentation instance collection data. Step S2: Based on the data collected from the segmentation instances, spatial features are extracted from the instances to be segmented, and the extraction results are analyzed for consistency with the scene objects in the target space scene. Based on the analysis results, the instance segmentation model is obtained, and scene instance segmentation data is obtained. Step S3: Verify the location rationality of the instance segmentation model in the target space scene based on the scene instance segmentation data, and issue an instance location warning based on the verification results.

[0006] Furthermore, in step S1, the specific steps are as follows: Step S11: Obtain the spatial scene that needs to be segmented into instances, and arbitrarily select a target spatial scene from the obtained spatial scenes; Step S12: Real-time acquisition of spatial audio appearing in the target spatial scene to obtain real-time spatial audio; extraction of continuous audio containing preset instruction keywords from the real-time spatial audio to obtain multiple keyword-extracted audio segments. Step S13: Randomly select one sample audio segment from the acquired multiple keyword-extracted audio segments, perform audio index analysis on the sample audio segment, and classify the sample audio segment into different types based on the analysis results. Step S14: Use keywords other than the sample extracted audio to extract audio to replace the sample extracted audio, and classify the replaced sample extracted audio by type to obtain multiple valid extracted audio. Step S15: Use an audio recognition algorithm to extract the segments to be segmented contained in the valid audio clips to obtain segmentation instance collection data.

[0007] Furthermore, in step S13, the specific steps are as follows: The audio sample is segmented into short frames of a preset time length to obtain multiple sample audio short frames. MFCC feature vectors are extracted from each sample audio short frame to obtain multiple sets of MFCC feature vectors. The average of the multiple sets of MFCC feature vectors is calculated to obtain the sample audio MFCC feature vector. Multiple audio samples are extracted from the effective audio segments generated by the target space scene at historical moments. A feature audio sample is randomly selected from the extracted audio samples and divided into short frames of a preset time length to obtain multiple feature audio short frames. MFCC feature vectors are extracted from each feature audio short frame to obtain multiple sets of MFCC feature vectors. The average of the obtained multiple sets of MFCC feature vectors is calculated to obtain the feature audio MFCC feature vector.

[0008] Furthermore, in step S13, the specific steps are as follows: The cosine similarity between the MFCC feature vector of the sample audio and the MFCC feature vector of the feature audio is calculated to obtain the cosine similarity between the MFCC feature vectors of the sample audio and the feature audio sample. A preset range of cosine similarity is set. If the cosine similarity of the MFCC feature vector is within the preset range, the sample audio is directly classified as a valid audio. If the cosine similarity of the MFCC feature vector is not within the preset range, the feature audio sample is replaced with an audio sample other than the feature audio sample. The above comparison process is repeated until the cosine similarity of the MFCC feature vector is within the preset range, or all audio samples have replaced the feature audio sample.

[0009] Furthermore, in step S2, the specific steps are as follows: Step S21: Obtain scene instance segmentation data, obtain the instance to be segmented based on the scene instance segmentation data, collect the historical instance graphics corresponding to the instance to be segmented, and obtain the historical graphics of the segmented instance. Step S22: Use a 3D camera to perform 3D modeling of the target space scene to obtain a target area space model, and collect the instance graphics covered in the target area space model to obtain multiple spatial instance graphics; Step S23: Randomly select a feature instance image from the acquired spatial instance images, compare the feature instance image with the segmentation instance historical images, and obtain the comparison consistency between the feature instance image and the segmentation instance historical images based on the comparison results. Step S24: Use the alignment consistency of each spatial instance image (excluding feature instance images) with the historical images of segmented instances, perform numerical comparison on the obtained multiple alignment consistency, and set the spatial instance image corresponding to the largest alignment consistency as the instance segmentation model corresponding to the instance to be segmented, to obtain scene instance segmentation data.

[0010] Furthermore, in step S23, the specific steps are as follows: The geometric center point corresponding to the feature instance graphic is collected to obtain the first graphic feature point. The graphic vertices contained in the feature instance graphic are acquired to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The length values ​​of the obtained multiple graphic vertex connection lines are compared. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second graphic feature point and the third graphic feature point, respectively. The two-dimensional closed graphic composed of the first graphic feature point, the second graphic feature point, and the third graphic feature point is marked as the first comparison two-dimensional graphic. The geometric center point corresponding to the segmented instance historical graphic is collected to obtain the first historical feature point. The graphic vertices contained in the segmented instance historical graphic are obtained to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The length values ​​of the obtained multiple graphic vertex connection lines are compared. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second historical feature point and the third historical feature point, respectively. The two-dimensional closed graphic formed by the first historical feature point, the second historical feature point, and the third historical feature point is marked as the second comparison two-dimensional graphic.

[0011] Furthermore, in step S23, the specific steps are as follows: The segmented instance historical image is spatially traversed in the target region spatial model. If the first comparison two-dimensional image and the second comparison two-dimensional image can overlap, the volume value of the overlapping image region of the segmented instance historical image and the feature instance image is collected to obtain the volume value of the overlapping image. The volume of the feature instance image is collected to obtain the volume value of the feature image. The ratio of the volume value of the overlapping image to the volume value of the feature image is calculated to obtain the matching consistency between the feature instance image and the segmented instance historical image. If the first and second comparison two-dimensional graphics cannot overlap, then the parameter is assigned to the alignment consistency between the feature instance graphics and the segmentation instance historical graphics using preset feature values.

[0012] Furthermore, in step S3, the specific steps are as follows: Step S31: Obtain scene instance segmentation data, and obtain the instance segmentation model based on the scene instance segmentation data; Step S32: Obtain the spatial model corresponding to the target space scene to obtain the target area spatial model, and create a spatial coordinate system in the target area spatial model to obtain the target space three-dimensional coordinate system; Step S33: Spatial acquisition is performed on the preset deployment position of the instance segmentation model in the target area spatial model to obtain the preset position spatial volume. The preset position spatial volume is discretized into several three-dimensional spatial points. The three-dimensional coordinates corresponding to each three-dimensional spatial point are obtained according to the target space three-dimensional coordinate system to obtain the preset position coordinate set.

[0013] Furthermore, in step S3, the specific steps are as follows: Step S34: Obtain the actual space occupied by the instance segmentation model in the target region spatial model to obtain the actual position space volume. Discretize the actual position space volume into several three-dimensional space points. Obtain the three-dimensional coordinates corresponding to each three-dimensional space point according to the target space three-dimensional coordinate system to obtain the three-dimensional coordinates of multiple instance geometric points. Step S35: If the three-dimensional coordinates of all instance geometric points are included in the preset position coordinate set, the instance segmentation model is judged to be in a reasonable position in the target space scene. If the three-dimensional coordinates of any instance geometric point are not included in the preset position coordinate set, the instance segmentation model is judged to be in an unreasonable position in the target space scene, and an instance position warning is issued for the instance segmentation model.

[0014] A multimodal fusion spatial intelligent scene instance segmentation system includes: Data acquisition module: Collects audio for keyword extraction in the target space scene, extracts valid audio from the keyword-extracted audio through audio index analysis, and collects segmentation instance data from the valid audio to be segmented. Instance segmentation module: Based on the data collected from the segmentation instances, it extracts spatial features from the instances to be segmented, performs consistency analysis between the extraction results and scene objects in the target space scene, obtains the instance segmentation model based on the analysis results, and obtains scene instance segmentation data; Location verification module: Verifies the location rationality of the instance segmentation model in the target space scene based on the scene instance segmentation data, and issues instance location warnings based on the verification results.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention performs keyword-based audio capture for target spatial scenes, extracts effective audio from the keyword-based audio through audio index analysis, and collects instances to be segmented from the effective audio. This can more accurately capture effective information related to the instances to be segmented within the target spatial scene, improve the ability to identify and analyze instances in the scene, and thus enhance the performance and reliability of the entire intelligent scene instance segmentation system. 2. This invention extracts spatial features from the segmentation instance based on the data collected from the segmentation instance, performs consistency analysis between the extraction results and scene objects in the target space scene, obtains an instance segmentation model based on the analysis results, verifies the location rationality of the instance segmentation model in the target space scene, and provides instance location warnings based on the verification results, which can effectively ensure the accuracy and reliability of scene instance segmentation. Attached Figure Description

[0016] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0017] Figure 1 This is a diagram illustrating the implementation steps of the present invention; Figure 2 This is an overall system block diagram of the present invention; Figure 3 This is a schematic diagram of the first comparative two-dimensional graphic in this invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 Please see Figure 1 This invention provides a technical solution: a multimodal fusion spatial intelligent scene instance segmentation method, the specific steps of which are as follows: Step S1: Collect keyword-trimmed audio for the target space scene, extract the effective trimmed audio from the keyword-trimmed audio through audio index analysis, and collect segmentation instances from the effective trimmed audio to obtain segmentation instance collection data. In step S1, the specific steps are as follows: Step S11: Obtain the spatial scene that needs to be segmented into instances, and arbitrarily select a target spatial scene from the obtained spatial scenes; Step S12: Real-time acquisition of spatial audio appearing in the target spatial scene to obtain real-time spatial audio; extraction of continuous audio containing preset instruction keywords from the real-time spatial audio to obtain multiple keyword-extracted audio segments. Step S13: Randomly select one sample audio segment from the acquired multiple keyword-extracted audio segments, perform audio index analysis on the sample audio segment, and classify the sample audio segment into different types based on the analysis results. In step S13, the specific steps are as follows: The audio sample is segmented into short frames of a preset time length to obtain multiple sample audio short frames. MFCC feature vectors are extracted from each sample audio short frame to obtain multiple sets of MFCC feature vectors. The average of the multiple sets of MFCC feature vectors is calculated to obtain the sample audio MFCC feature vector. From the effective audio segments generated by the target space scene at historical moments, multiple sets of audio samples are extracted. A feature audio sample is randomly selected from the extracted audio samples and divided into short frames of a preset time length to obtain multiple feature audio short frames. MFCC feature vectors are extracted from each feature audio short frame to obtain multiple sets of MFCC feature vectors. The average of the obtained multiple sets of MFCC feature vectors is calculated to obtain the feature audio MFCC feature vector. In step S13, the specific steps are as follows: The cosine similarity between the sample audio MFCC feature vector and the feature audio MFCC feature vector is calculated to obtain the cosine similarity between the sample audio and the feature audio sample. A preset range of cosine similarity is set. If the cosine similarity of the MFCC feature vector is within the preset range, the sample audio is directly classified as a valid audio. If the cosine similarity of the MFCC feature vector is not within the preset range, the feature audio sample is replaced with an audio sample other than the feature audio sample. The above comparison process is repeated until the cosine similarity of the MFCC feature vector is within the preset range, or all audio samples have replaced the feature audio sample. Step S14: Use keywords other than the sample extracted audio to extract audio to replace the sample extracted audio, and classify the replaced sample extracted audio by type to obtain multiple valid extracted audio. Step S15: Use an audio recognition algorithm to extract the segments to be segmented contained in the valid audio clips to obtain segmentation instance collection data; Step S2: Based on the data collected from the segmentation instances, spatial features are extracted from the instances to be segmented, and the extraction results are analyzed for consistency with the scene objects in the target space scene. Based on the analysis results, the instance segmentation model is obtained, and scene instance segmentation data is obtained. In step S2, the specific steps are as follows: Step S21: Obtain scene instance segmentation data, obtain the instance to be segmented based on the scene instance segmentation data, collect the historical instance graphics corresponding to the instance to be segmented, and obtain the historical graphics of the segmented instance. Step S22: Use a 3D camera to perform 3D modeling of the target space scene to obtain a target area space model, and collect the instance graphics covered in the target area space model to obtain multiple spatial instance graphics; Step S23: Randomly select a feature instance image from the acquired spatial instance images, compare the feature instance image with the segmentation instance historical images, and obtain the comparison consistency between the feature instance image and the segmentation instance historical images based on the comparison results. In step S23, the specific steps are as follows: The geometric center point corresponding to the feature instance graphic is collected to obtain the first graphic feature point. The graphic vertices contained in the feature instance graphic are acquired to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The length values ​​of the obtained multiple graphic vertex connection lines are compared. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second graphic feature point and the third graphic feature point, respectively. The two-dimensional closed graphic composed of the first graphic feature point, the second graphic feature point, and the third graphic feature point is marked as the first comparison two-dimensional graphic. The geometric center point corresponding to the segmented instance historical graphic is collected to obtain the first historical feature point. The graphic vertices contained in the segmented instance historical graphic are obtained to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The length values ​​of the obtained multiple graphic vertex connection lines are compared. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second historical feature point and the third historical feature point, respectively. The two-dimensional closed graphic composed of the first historical feature point, the second historical feature point, and the third historical feature point is marked as the second comparison two-dimensional graphic. In step S23, the specific steps are as follows: The segmented instance historical image is spatially traversed in the target region spatial model. If the first comparison two-dimensional image and the second comparison two-dimensional image can overlap, the volume value of the overlapping image region of the segmented instance historical image and the feature instance image is collected to obtain the volume value of the overlapping image. The volume of the feature instance image is collected to obtain the volume value of the feature image. The ratio of the volume value of the overlapping image to the volume value of the feature image is calculated to obtain the matching consistency between the feature instance image and the segmented instance historical image. If the first comparison two-dimensional graphic and the second comparison two-dimensional graphic cannot be superimposed, then the parameter value is assigned to the comparison consistency between the feature instance graphic and the segmentation instance historical graphic using the preset feature value. Step S24: Use the alignment consistency of each spatial instance image (excluding feature instance images) with the historical images of segmented instances, perform numerical comparison on the multiple alignment consistency results, and set the spatial instance image corresponding to the highest alignment consistency as the instance segmentation model corresponding to the instance to be segmented, to obtain scene instance segmentation data. Step S3: Verify the reasonableness of the location of the instance segmentation model in the target space scene based on the scene instance segmentation data, and issue an instance location warning based on the verification results; The specific steps in step S3 are as follows: Step S31: Obtain scene instance segmentation data, and obtain the instance segmentation model based on the scene instance segmentation data; Step S32: Obtain the spatial model corresponding to the target space scene to obtain the target area spatial model, and create a spatial coordinate system in the target area spatial model to obtain the target space three-dimensional coordinate system; Step S33: Spatial acquisition is performed on the preset deployment position of the instance segmentation model in the target area spatial model to obtain the preset position spatial volume. The preset position spatial volume is discretized into several three-dimensional spatial points. The three-dimensional coordinates corresponding to each three-dimensional spatial point are obtained according to the target space three-dimensional coordinate system to obtain the preset position coordinate set. The specific steps in step S3 are as follows: Step S34: Obtain the actual space occupied by the instance segmentation model in the target region spatial model to obtain the actual position space volume. Discretize the actual position space volume into several three-dimensional space points. Obtain the three-dimensional coordinates corresponding to each three-dimensional space point according to the target space three-dimensional coordinate system to obtain the three-dimensional coordinates of multiple instance geometric points. Step S35: If the three-dimensional coordinates of all instance geometric points are included in the preset position coordinate set, the instance segmentation model is judged to be in a reasonable position in the target space scene. If the three-dimensional coordinates of any instance geometric point are not included in the preset position coordinate set, the instance segmentation model is judged to be in an unreasonable position in the target space scene, and an instance position warning is issued for the instance segmentation model.

[0020] Example 2 Please see Figure 2 Based on another concept of the same invention, a multimodal fusion spatial intelligent scene instance segmentation system is proposed. The specific working process of each module is as follows: The data acquisition module collects keyword-truncation audio for the target spatial scene, extracts the effective truncated audio from the keyword-truncation audio through audio index analysis, and collects segmentation instance data from the effective truncated audio. Specifically as follows: Acquire the spatial scenes that need to be segmented into instances, and arbitrarily select a target spatial scene from the acquired spatial scenes; It should be noted here that: In this application, all spatial scenes involved are equipped with a spatial intelligent scene instance segmentation system.

[0021] Real-time acquisition of spatial audio in the target spatial scene is performed to obtain real-time spatial audio. Continuous audio containing preset instruction keywords in the real-time spatial audio is extracted to obtain multiple segments of keyword-extracted audio. It should be noted here that: In this application, the preset instruction keywords involved include, but are not limited to, “start instance splitting”, “stop instance splitting”, and “continue instance splitting”. In practical applications, the preset instruction keywords can be manually set.

[0022] Select any one sample audio segment from the acquired multiple keyword-extracted audio segments, perform audio index analysis on the sample audio segment, and classify the sample audio segment into different types based on the analysis results. Specifically as follows: The audio sample is segmented into short frames of a preset time length to obtain multiple sample audio short frames. MFCC feature vectors are extracted from each sample audio short frame to obtain multiple sets of MFCC feature vectors. The average of the multiple sets of MFCC feature vectors is calculated to obtain the sample audio MFCC feature vector. It should be noted here that: In this application, the MFCC feature vector involved here is specifically composed of 12-dimensional basic MFCC + 12-dimensional first-order difference ΔMFCC, forming a 24-dimensional feature vector; In this application, the preset time length referred to herein is specifically 20ms.

[0023] From the effective audio segments generated by the target space scene at historical moments, multiple sets of audio samples are extracted. A feature audio sample is randomly selected from the extracted audio samples and divided into short frames of a preset time length to obtain multiple feature audio short frames. MFCC feature vectors are extracted from each feature audio short frame to obtain multiple sets of MFCC feature vectors. The average of the obtained multiple sets of MFCC feature vectors is calculated to obtain the feature audio MFCC feature vector. It should be noted here that: In this application, the valid audio capture referred to herein is the human voice audio generated by the preset command issuer.

[0024] The cosine similarity between the sample audio MFCC feature vector and the feature audio MFCC feature vector is calculated to obtain the cosine similarity between the sample audio and the feature audio sample. A preset range of cosine similarity is set. If the cosine similarity of the MFCC feature vector is within the preset range, the sample audio is directly classified as a valid audio. If the cosine similarity of the MFCC feature vector is not within the preset range, the feature audio sample is replaced with an audio sample other than the feature audio sample. The above comparison process is repeated until the cosine similarity of the MFCC feature vector is within the preset range, or all audio samples have replaced the feature audio sample. It should be noted here that: MFCC feature vector cosine similarity is collected for the valid audio segments obtained from historical comparisons. The cosine similarity of the obtained historical feature vectors is numerically compared. The historical feature vector with the largest value is set as the upper limit of the preset cosine similarity interval, and the historical feature vector with the smallest value is set as the lower limit of the preset cosine similarity interval. The audio was replaced using keywords other than the extracted audio samples, and the replaced audio samples were then categorized by type to obtain multiple valid audio samples. The audio recognition algorithm is used to extract the segments to be segmented contained in the valid audio clips, and the segmentation segment data is obtained. The instance segmentation module extracts spatial features of the instance to be segmented based on the data collected from the segmentation instances, performs consistency analysis between the extraction results and scene objects in the target space scene, obtains the instance segmentation model based on the analysis results, and obtains scene instance segmentation data. Obtain scene instance segmentation data, obtain the instance to be segmented based on the scene instance segmentation data, collect the historical instance graphics corresponding to the instance to be segmented, and obtain the historical graphics of the segmented instance. It should be noted here that: In this application, the segmentation instance history graph specifically refers to the graph of the segmentation record of the instance to be segmented in the historical segmentation record, and the segmentation instance history graph involved here is specifically a three-dimensional model.

[0025] A 3D camera is used to create a 3D model of the target space scene to obtain a spatial model of the target area. The instance graphics covered in the spatial model of the target area are then captured to obtain multiple spatial instance graphics. Arbitrarily select a feature instance image from the acquired spatial instance images, compare the feature instance image with the segmentation instance historical images, and obtain the comparison consistency between the feature instance image and the segmentation instance historical images based on the comparison results. Specifically as follows: Please see Figure 3 The geometric center point corresponding to the feature instance graphic is collected to obtain the first graphic feature point. The graphic vertices contained in the feature instance graphic are obtained to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The length values ​​of the obtained multiple graphic vertex connection lines are compared. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second graphic feature point and the third graphic feature point, respectively. The two-dimensional closed graphic composed of the first graphic feature point, the second graphic feature point, and the third graphic feature point is marked as the first comparison two-dimensional graphic. The geometric center point corresponding to the segmented instance historical graphic is collected to obtain the first historical feature point. The graphic vertices contained in the segmented instance historical graphic are obtained to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The length values ​​of the obtained multiple graphic vertex connection lines are compared. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second historical feature point and the third historical feature point, respectively. The two-dimensional closed graphic composed of the first historical feature point, the second historical feature point, and the third historical feature point is marked as the second comparison two-dimensional graphic. The segmented instance historical image is spatially traversed in the target region spatial model. If the first comparison two-dimensional image and the second comparison two-dimensional image can overlap, the volume value of the overlapping image region of the segmented instance historical image and the feature instance image is collected to obtain the volume value of the overlapping image. The volume of the feature instance image is collected to obtain the volume value of the feature image. The ratio of the volume value of the overlapping image to the volume value of the feature image is calculated to obtain the matching consistency between the feature instance image and the segmented instance historical image. If the first comparison two-dimensional graphic and the second comparison two-dimensional graphic cannot be superimposed, then the parameter value is assigned to the comparison consistency between the feature instance graphic and the segmentation instance historical graphic using the preset feature value. It should be noted here that: In this application, the preset feature value referred to herein is specifically 0.

[0026] The alignment consistency between each spatial instance image (excluding the feature instance image) and the historical images of the segmented instances is used to perform numerical comparison on the multiple alignment consistency results. The spatial instance image corresponding to the highest alignment consistency is set as the instance segmentation model corresponding to the instance to be segmented, thus obtaining the scene instance segmentation data. The location verification module verifies the location rationality of the instance segmentation model in the target space scene based on the scene instance segmentation data, and issues instance location warnings based on the verification results. Specifically as follows: Obtain scene instance segmentation data, and then obtain the instance segmentation model based on the scene instance segmentation data; The spatial model corresponding to the target space scene is acquired to obtain the target area spatial model, and a spatial coordinate system is created in the target area spatial model to obtain the target space three-dimensional coordinate system; Spatial acquisition is performed on the preset deployment location of the instance segmentation model in the target area spatial model to obtain the preset location spatial volume. The preset location spatial volume is discretized into several three-dimensional spatial points. The three-dimensional coordinates corresponding to each three-dimensional spatial point are obtained according to the target space three-dimensional coordinate system to obtain the preset location coordinate set. The actual space occupied by the instance segmentation model in the target region spatial model is obtained to obtain the actual location space volume. The actual location space volume is discretized into several three-dimensional space points. The three-dimensional coordinates corresponding to each three-dimensional space point are obtained according to the target space three-dimensional coordinate system to obtain the three-dimensional coordinates of multiple instance geometric points. If all three-dimensional coordinates of the instance geometric points are included in the preset location coordinate set, the instance segmentation model is judged to be in a reasonable position in the target space scene. If any instance geometric point has three-dimensional coordinates that are not included in the preset location coordinate set, the instance segmentation model is judged to be in an unreasonable position in the target space scene, and an instance position warning is issued for the instance segmentation model.

[0027] Compared to the problems described in the background technology, the present invention performs keyword-based audio capture of target spatial scenes, extracts effective audio from the keyword-based audio through audio index analysis, and collects instances to be segmented from the effective audio. This can more accurately capture effective information related to the instances to be segmented within the target spatial scene, improve the ability to identify and analyze instances in the scene, and thus enhance the performance and reliability of the entire intelligent spatial scene instance segmentation system. Furthermore, this invention extracts spatial features from the segmentation instance data, analyzes the consistency between the extracted features and scene objects in the target space scene, obtains an instance segmentation model based on the analysis results, verifies the location rationality of the instance segmentation model in the target space scene, and provides instance location warnings based on the verification results. This effectively ensures the accuracy and reliability of scene instance segmentation. In spatial intelligent scenarios, it ensures a high degree of fit between the instance segmentation model and the actual scene.

[0028] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal fusion spatial intelligent scene instance segmentation method, characterized in that, include: Step S1: Collect audio for keyword extraction in the target space scene, extract the effective extracted audio from the keyword extracted audio through audio index analysis, collect the segmentation instance data from the effective extracted audio; Step S2: Based on the data collected from the segmentation instances, spatial features are extracted from the instances to be segmented, and the extraction results are analyzed for consistency with the scene objects in the target space scene. Based on the analysis results, the instance segmentation model is obtained, and scene instance segmentation data is obtained. In step S2, the specific steps are as follows: Step S21: Obtain scene instance segmentation data, obtain the instance to be segmented based on the scene instance segmentation data, collect the historical instance graphics corresponding to the instance to be segmented, and obtain the historical graphics of the segmented instance. Step S22: Perform 3D modeling of the target space scene to obtain the target area space model, and collect the instance graphics contained in the target area space model to obtain multiple spatial instance graphics; Step S23: Randomly select a feature instance image from the acquired spatial instance images, compare the feature instance image with the segmentation instance historical images, and obtain the comparison consistency between the feature instance image and the segmentation instance historical images. Step S24: Obtain the alignment consistency between different spatial instance graphics and historical segmentation instance graphics, and set the spatial instance graphics corresponding to the highest alignment consistency as the instance segmentation model to obtain scene instance segmentation data; In step S23, the specific steps are as follows: The geometric center point corresponding to the feature instance graphic is collected to obtain the first graphic feature point. The graphic vertices contained in the feature instance graphic are obtained to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second graphic feature point and the third graphic feature point, respectively. The two-dimensional closed graphic composed of the first graphic feature point, the second graphic feature point, and the third graphic feature point is marked as the first comparison two-dimensional graphic. The geometric center point corresponding to the segmented instance historical graphic is collected to obtain the first historical feature point. The graphic vertices contained in the segmented instance historical graphic are obtained to obtain multiple feature graphic vertices. Any two feature graphic vertices are connected to obtain multiple graphic vertex connection lines. The graphic vertex connection line corresponding to the maximum length value is marked as the major axis vertex connection line. The two graphic vertices that make up the major axis vertex connection line are named the second historical feature point and the third historical feature point, respectively. The two-dimensional closed graphic composed of the first historical feature point, the second historical feature point, and the third historical feature point is marked as the second comparison two-dimensional graphic. The segmented instance historical image is spatially traversed in the target region spatial model. If the first comparison two-dimensional image and the second comparison two-dimensional image can overlap, the volume of the overlapping image region of the segmented instance historical image and the feature instance image is collected to obtain the overlapping image volume value. The volume of the feature instance image is collected to obtain the feature image volume value. The ratio of the overlapping image volume value to the feature image volume value is calculated to obtain the comparison consistency. If the first and second two-dimensional comparison graphics cannot be superimposed, the consistency parameter is assigned using the preset feature value comparison. Step S3: Verify the location rationality of the instance segmentation model in the target space scene based on the scene instance segmentation data, and issue an instance location warning based on the verification results.

2. The spatial intelligent scene instance segmentation method based on multimodal fusion according to claim 1, characterized in that, In step S1, the specific steps are as follows: Step S11: Obtain the spatial scene that needs to be segmented into instances, and arbitrarily select a target spatial scene from the obtained spatial scenes; Step S12: Real-time acquisition of spatial audio appearing in the target spatial scene to obtain real-time spatial audio; extraction of continuous audio containing preset instruction keywords from the real-time spatial audio to obtain multiple keyword-extracted audio segments. Step S13: Select a sample audio segment from the acquired multiple keyword-extracted audio segments, perform audio index analysis on the sample audio segment, and classify the sample audio segment by type. Step S14: Use keywords other than the sample extracted audio to extract audio to replace the sample extracted audio, and classify the replaced sample extracted audio by type to obtain multiple valid extracted audio. Step S15: Use an audio recognition algorithm to extract the segments to be segmented contained in the valid audio clips to obtain segmentation instance collection data.

3. The spatial intelligent scene instance segmentation method based on multimodal fusion according to claim 2, characterized in that, In step S13, the specific steps are as follows: The sample audio is segmented into short frames of a preset time length to obtain multiple sample audio short frames. MFCC feature vectors are extracted from each sample audio short frame to obtain multiple sets of MFCC feature vectors. The average of these MFCC feature vectors is then calculated to obtain the sample audio MFCC feature vector. Feature audio samples are collected from the effective audio segments generated by the target spatial scene at historical moments. The feature audio samples are divided into short frames of a preset time length to obtain multiple feature audio short frames. MFCC feature vectors are extracted from each feature audio short frame to obtain multiple sets of MFCC feature vectors. The average of these sets is then calculated to obtain the feature audio MFCC feature vector.

4. The spatial intelligent scene instance segmentation method based on multimodal fusion according to claim 3, characterized in that, In step S13, the specific steps are as follows: The cosine similarity between the sample audio MFCC feature vector and the feature audio MFCC feature vector is calculated to obtain the cosine similarity of the MFCC feature vector. A preset interval for the cosine similarity is set. If the cosine similarity of the MFCC feature vector is within the preset interval, the sample audio is directly classified as valid audio. If it is not within the preset interval, the feature audio sample is replaced with audio samples other than the feature audio sample, and the interval comparison process is repeated until the cosine similarity of the MFCC feature vector is within the preset interval, or the replacement of the feature audio sample with all audio samples is completed.

5. The spatial intelligent scene instance segmentation method based on multimodal fusion according to claim 1, characterized in that, The specific steps in step S3 are as follows: Step S31: Obtain scene instance segmentation data, and obtain the instance segmentation model based on the scene instance segmentation data; Step S32: Obtain the spatial model corresponding to the target space scene to obtain the target area spatial model, and create a spatial coordinate system in the target area spatial model to obtain the target space three-dimensional coordinate system; Step S33: Spatial acquisition is performed on the preset deployment position of the instance segmentation model in the target area spatial model to obtain the preset position spatial volume. The preset position spatial volume is discretized into several three-dimensional spatial points. The three-dimensional coordinates corresponding to each three-dimensional spatial point are obtained according to the target space three-dimensional coordinate system to obtain the preset position coordinate set.

6. The spatial intelligent scene instance segmentation method based on multimodal fusion according to claim 5, characterized in that, The specific steps in step S3 are as follows: Step S34: Obtain the spatial region corresponding to the target region spatial model of the instance segmentation model to obtain the actual position spatial volume, discretize the actual position spatial volume into several three-dimensional spatial points, and obtain the three-dimensional coordinates corresponding to each three-dimensional spatial point according to the target space three-dimensional coordinate system to obtain the three-dimensional coordinates of multiple instance geometric points. Step S35: If the three-dimensional coordinates of all instance geometric points are included in the preset position coordinate set, the instance segmentation model is judged to be in a reasonable position in the target space scene. If the three-dimensional coordinates of any instance geometric point are not included in the preset position coordinate set, the instance segmentation model is judged to be in an unreasonable position in the target space scene, and an instance position warning is issued for the instance segmentation model.

7. A multimodal fusion spatial intelligent scene instance segmentation system, applicable to the multimodal fusion spatial intelligent scene instance segmentation method according to any one of claims 1-6, characterized in that, The instance segmentation system includes: Data acquisition module: Collects audio for keyword extraction in the target space scene, extracts valid audio from the keyword-extracted audio through audio index analysis, and collects segmentation instance data from the valid audio to be segmented. Instance segmentation module: Based on the data collected from the segmentation instances, it extracts spatial features of the instances to be segmented, performs consistency analysis between the extraction results and scene objects in the target space scene, obtains the instance segmentation model, and gets the scene instance segmentation data; Location verification module: Verifies the location rationality of the instance segmentation model in the target space scene based on the scene instance segmentation data, and issues instance location warnings based on the verification results.

Images