A wartime field intelligent rescue system

Abstract

The embodiment of the present application provides a wartime field intelligent rescue system, the system includes a wartime module, which is used for soldiers to identify and retrieve through keywords, text, images and / or voice in a multimodal combat injury database to retrieve and match combat injury information, and obtain accurate treatment plans based on the retrieved and matched information; a training module, the training module includes training parameter settings, the training parameter settings are used to autonomously select training parameters, and the training parameters include combat environment and combat injury categories; virtual combat injury treatment, virtual combat injury treatment is used to treat combat injuries for virtual people in the training parameters. The embodiment of the present application can improve the survival rate of the wounded under emergency conditions in wartime, intelligently identify the injuries based on the collection of voice and image data of the wounded on the spot, and select emergency treatment measures and treatment steps to guide the wounded to complete the first aid operation autonomously or with mutual assistance.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of field intelligent rescue, and in particular, to a wartime field intelligent rescue system. Background Art

[0002] Military training is an activity of military theory education and combat skills training, which is usually divided into troop training, school training and reserve training. Military training is the main task of the army in peacetime and an important aspect of combat readiness. In order to improve the actual combat capability of the army, military exercises are often conducted in large military venues. Military exercises sometimes last for several days, which seriously tests the continuous combat capability of soldiers. However, long-term military training exercises are bound to lead to accidents and emergencies, which require rescue.

[0003] However, the existing rescue methods for large-scale military training exercises have the following problems: there is a lack of a rescue system specifically designed for large-scale military training and exercises, and rescue is often delayed in actual operation. Therefore, it is necessary to design corresponding technical solutions to solve the existing technical problems. Summary of the invention

[0004] The purpose of the embodiment of the present application is to provide a wartime field intelligent rescue system, which uses multiple forms such as keyword text, image and / or voice for recognition and retrieval to accurately and intuitively display them to inexperienced non-professionals, guide them to carry out emergency treatment of typical war trauma, and improve the survival rate of the wounded under wartime emergency conditions. It can intelligently identify the injuries based on the collection of on-site voice and image data of the wounded, and select emergency treatment measures and treatment steps, guide the wounded to complete the first aid operation independently or with mutual assistance, and record relevant image data at the same time, and automatically match and reconstruct virtual reality skill training cases for emergency treatment of battlefield injuries after the war, so as to simulate actual combat conditions to the greatest extent and improve the effectiveness of daily training, thereby solving the problems in the background technology.

[0005] In order to solve the above technical problems, the technical solution of the wartime field intelligent rescue system provided by the embodiment of the present application is as follows:

[0006] The embodiment of the present application discloses a wartime field intelligent rescue system, the system comprising:

[0007] The wartime module is used for soldiers to identify and search for combat injury information in the multimodal combat injury database through keyword text, image and / or voice, so as to retrieve and match combat injury information and obtain accurate treatment plans based on the retrieved and matched information;

[0008] If it is during combat, the collected image, voice, and text data will be automatically saved in the multimodal battle injury database, and after the data collected by the smart interactive glasses is recovered to the military headquarters, the data will be transmitted to the multimodal battle injury database for intelligent situational image analysis;

[0009] The training module includes training parameter settings, which are used to autonomously select training parameters, including combat environment and combat injury categories; virtual combat injury treatment, which is used to treat combat injuries of virtual people in the training parameters.

[0010] In a preferred embodiment of any of the above schemes, the virtual combat injury treatment further includes the following steps:

[0011] By setting the training parameters, the battle injury scene, battle injury category and location are virtually constructed on the virtual human;

[0012] Soldiers perform simulated combat injury treatment operations on virtual humans according to the treatment tools provided by the training parameter settings, wherein the simulated combat injury treatment operations include scaling or rotating the virtual humans, controlling and accurately locating combat injury details and implementing treatment.

[0013] In a preferred embodiment of any of the above schemes, the intelligent scene image analysis comprises the following steps:

[0014] Step 1: Find a point P in the space, find the n points closest to point P through KdTree, calculate the distance from n points to point P, and classify the points whose distance is less than the threshold into the Q category;

[0015] Step 2: Find unsearched points in class Q and repeat step 1.

[0016] Step 3: The search is completed until all points in the Q class are searched. After the plane point cloud in the scene is segmented, the object point clouds are extracted one by one. The scene objects are segmented by setting thresholds through the Euclidean clustering method, and objects of different point cloud scales in the scene are extracted.

[0017] In a preferred embodiment of any of the above schemes, the intelligent scene image analysis further includes the following steps:

[0018] Step 31: After the object point cloud is segmented, the object point cloud group is projected onto the 2D plane by calculation, and the pixel coordinates of each point are calculated as:

[0019] clound.point[i].pixel_x=clound.indices[i]%width;

[0020] clound.point[i].pixel_y = clound.indices[i] / width, where Point[i].pixel_x is the x coordinate, Point[i].pixel_y is the y coordinate, Indices[i] is the pixel index value, and Width is the image width;

[0021] Step 32: After the object point cloud is projected to the pixel coordinates, the 2D bounding box of the object point cloud in the pixel coordinates is obtained by comparing the maximum and minimum values ​​of the point cloud in the pixel coordinate axis direction;

[0022] Step 33: perform matching similarity calculation between the back-projected two-dimensional bounding box of the object point cloud and the circumscribed rectangular bounding box output by the target detection algorithm, record the object point cloud that is most similar to the object identified by the target detection, generate a labeled 3D target object virtual object model, and superimpose the virtual object model and the real environment, wherein the similarity calculation uses IOU, border center distance and size difference as calculation indicators for matching similarity.

[0023] In a preferred embodiment of any of the above schemes, the IOU is used to represent the overlap rate between the target area generated by the model and the original marked real area, that is, the intersection of the detection frame and the real frame is divided by the union of the detection frame and the real frame, and the calculation formula is:

[0024]

[0025] In a preferred embodiment of any of the above schemes, the calculation method of the frame center distance includes the following steps:

[0026] The distance between the centers of the two borders P1 and P2 is used to describe the distance similarity, and the size difference is used to describe the size similarity. match Among them, x represents the horizontal coordinate of point p and the vertical coordinate of point p.

[0027] In a preferred embodiment of any of the above schemes, the superposition of the virtual object model and the real environment, the superposition method thereof comprises the following steps:

[0028] Step 331: extract the projected quadrilateral region and the feature points in the quadrilateral region from the video stream, adjust the size and position of the tracking window according to the current feature point position and movement direction, if the marker is visible, proceed to step 332; if the marker is not visible, but more than four non-collinear feature points in the area to be registered are tracked successfully, proceed to step 333, otherwise set the three-dimensional transformation matrix of the current frame to be the same as the previous frame, and proceed to step 334;

[0029] Step 332: The entire logo is visible, and the three-dimensional transformation matrix and the camera intrinsic parameter matrix are directly calculated based on the logo projection, and then the process goes to step 334;

[0030] Step 333: Calculate the current 3D transformation matrix using the homography relationship between planes and the transformation matrix of the previous frame. If the internal parameters of the camera do not change, proceed to step 334.

[0031] Step 334: Use the transformation matrix to achieve three-dimensional registration and go to step 331.

[0032] In a preferred embodiment of any of the above schemes, the superposition of the virtual object model and the real environment, the superposition method further comprises the following steps:

[0033] Step 3311: Ensure that each feature point is located at the intersection of multiple straight lines;

[0034] Step 3312: adopting a virtual-real registration method based on a global homography matrix, using the global homography matrix corresponding to the previous frame to transform the initial image, and performing feature matching between the transformed image and the current image;

[0035] Step 3313: Establish a homography relationship between the current frame and the initial frame.

[0036] In a preferred embodiment of any of the above schemes, the superposition of the virtual object model and the real environment, the superposition method further comprises the following steps:

[0037] Step 33121: extracting image feature points;

[0038] Step 33122: achieving effective matching between image feature points and local map points through image feature descriptors;

[0039] Step 33123: Calculate the map point reprojection error, use the bundle adjustment method to optimize the camera posture, and achieve camera tracking and positioning at the moment of each frame image acquisition.

[0040] In a preferred embodiment of any of the above schemes, in step 33123, if tracking and positioning fails, a key frame closest to the description of the current frame is found using a repositioning method based on a bag-of-words model, and feature points of the key frame and the current frame are matched and posture estimated.

[0041] Compared with the prior art, the wartime field intelligent rescue system of the embodiment of the present application uses multiple forms such as keyword text, image and / or voice for recognition and retrieval to accurately and intuitively display them to inexperienced non-professionals, guide them to carry out emergency treatment of typical war trauma, and improve the survival rate of the wounded under wartime emergency conditions. It can intelligently identify the injuries based on the collection of on-site voice and image data of the wounded, and select emergency treatment measures and treatment steps, guide the wounded to complete the first aid operation independently or with mutual assistance, and record relevant image data at the same time. After the war, it automatically matches and reconstructs virtual reality skill training cases for emergency treatment of battlefield injuries, so as to simulate actual combat conditions to the greatest extent and improve the effectiveness of daily training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a component of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. Some specific embodiments of the present application will be described in detail in an illustrative and non-restrictive manner with reference to the drawings. The same reference numerals in the drawings indicate the same or similar components or components. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the drawings:

[0043] Figure 1 This is a flow chart of the wartime field intelligent rescue system according to an embodiment of the present application.

[0044] Figure 2 This is a schematic diagram of the medical expert knowledge base of the wartime field intelligent rescue system in the embodiment of the present application.

[0045] Figure 3 This is a schematic diagram of the projection matching results of the wartime field intelligent rescue system of the embodiment of the present application.

[0046] Figure 4 This is a schematic diagram of the point cloud semantic fusion results of the wartime field intelligent rescue system in an embodiment of the present application.

[0047] Figure 5 This is a schematic diagram of the three-dimensional registration process of corner tracking under occlusion conditions of the wartime field intelligent rescue system in an embodiment of the present application.

[0048] Figure 6 This is a schematic diagram of the ORB-SLAM2-RGBD mode framework of the wartime field intelligent rescue system in an embodiment of the present application.

[0049] Figure 7 This is a schematic diagram of the virtual-reality fusion process of sparse point cloud based on semantic depth map of the wartime field intelligent rescue system in the embodiment of the present application.

[0050] Figure 8This is a flow chart of a method for superimposing a virtual object model and a real environment of a wartime field intelligent rescue system according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a component of the present application, rather than embodiments of the entire component. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0052] Example

[0053] like Figure 1 As shown, the embodiment of the present application provides a wartime field intelligent rescue system, the system comprising:

[0054] The wartime module is used for soldiers to identify and search for combat injury information in the multimodal combat injury database through keyword text, image and / or voice, so as to retrieve and match combat injury information and obtain accurate treatment plans based on the retrieved and matched information;

[0055] If it is during combat, the collected image, voice, and text data will be automatically saved in the multimodal battle injury database, and after the data collected by the smart interactive glasses is recovered to the military headquarters, the data will be transmitted to the multimodal battle injury database for intelligent situational image analysis;

[0056] The training module includes training parameter settings, which are used to autonomously select training parameters, including combat environment and combat injury categories; virtual combat injury treatment, which is used to treat combat injuries of virtual people in the training parameters.

[0057] In the wartime field intelligent rescue system described in the embodiment of the present invention, keywords, text, images and / or voice are used for recognition and retrieval, and various forms such as these are accurately and intuitively displayed to inexperienced non-professionals, so as to guide them in emergency treatment of typical war injuries and improve the survival rate of the wounded under wartime emergency conditions. The system can intelligently identify the injuries based on the collection of the on-site voice and image data of the wounded, select emergency treatment measures and treatment steps, guide the wounded to complete the first aid operation independently or with mutual assistance, and record relevant image data at the same time. After the war, it can automatically match and reconstruct virtual reality skill training cases for emergency treatment of battlefield injuries, so as to simulate actual combat conditions to the greatest extent and improve the effectiveness of daily training.

[0058] like Figure 1 and Figure 2 As shown, the virtual combat injury treatment further includes the following steps:

[0059] By setting the training parameters, the battle injury scene, battle injury category and location are virtually constructed on the virtual human;

[0060] Soldiers perform simulated combat injury treatment operations on virtual humans according to the treatment tools provided by the training parameter settings, wherein the simulated combat injury treatment operations include scaling or rotating the virtual humans, controlling and accurately locating combat injury details and implementing treatment.

[0061] In the wartime field intelligent rescue system described in the embodiment of the present invention, training parameters are set, and soldiers independently select training parameters, including combat environment, combat injury category, etc. The field environment can be selected from common different combat environments such as tropical rainforest combat and desert combat. At the same time, the field environment can also switch between day and night modes. The battlefield injury category can be selected according to the injury database established inside the system, and common injuries can be selected. At the same time, the wound location can be further set according to different combat injury categories; virtual combat injury treatment is mainly performed by soldiers on virtual people in the system. In the early stage, the combat injury scene, combat injury category and location have been virtually constructed on the virtual person through parameter setting. Soldiers can simulate combat injury treatment operations on the virtual person according to the treatment tools provided by the system, and can zoom, rotate and other operations on the virtual person to accurately locate the details of the combat injury and implement treatment. At the same time, the system also provides prompts for the treatment plan, which are presented to the soldiers in any form of pictures or texts to assist them in better combat injury treatment.

[0062] In the wartime field intelligent first aid system described in the embodiment of the present invention, the wartime field intelligent first aid system also includes a system treatment score, and the standard basis of the system score is multi-dimensional, that is, the system internally established environmental difficulty, action standard, completion time, tool use, assistance ability, and first aid principles are used to perform comprehensive and objective scoring, and the performance data is supported to be exported. Environmental difficulty is based on the assessment of different scene environments, which can evaluate the emergency response ability of trainees in harsh environments; action standard is based on the assessment of different injury treatment, which can evaluate the mastery of the trainees' own battlefield treatment techniques; completion time is based on the success rate of treatment being inversely proportional to the treatment time, which can evaluate the mastery of the trainees' battlefield treatment techniques. Tool use is based on the correct use of treatment tools to achieve the effect of treatment, which can evaluate the trainees' judgment ability and the mastery of battlefield treatment techniques; assistance ability is based on whether the cooperation between multiple people is in place, which can evaluate the trainees' analysis ability and teamwork ability; first aid principle is based on the principle of the order of battlefield treatment implementation, which evaluates the trainees' judgment ability, team assistance ability and battlefield treatment ability. Through multi-dimensional objective and subjective scoring, the trainees' battlefield rescue level and comprehensive capabilities are comprehensively evaluated.

[0063] In an embodiment of the present invention, the intelligent scene image analysis includes the following steps:

[0064] Step 1: Find a point P in the space, find the n points closest to point P through KdTree (short for k-dimensional tree), calculate the distance from n points to point P, and classify the points whose distance is less than the threshold into class Q;

[0065] Step 2: Find unsearched points in class Q and repeat step 1.

[0066] Step 3: The search is completed until all points in the Q class are searched. After the plane point cloud in the scene is segmented, the object point clouds are extracted one by one. The scene objects are segmented by setting thresholds through the Euclidean clustering method, and objects of different point cloud scales in the scene are extracted.

[0067] In the wartime field intelligent rescue system described in the embodiment of the present invention, the object point cloud can be quickly segmented by adopting the above analysis method.

[0068] like Figure 3 and Figure 4 As shown, the intelligent scene image analysis also includes the following steps:

[0069] Step 31: After the object point cloud is segmented, the object point cloud group is projected onto the 2D plane by calculation, and the pixel coordinates of each point are calculated as:

[0070] clound.point[i].pixel_x=clound.indices[i]%width;

[0071] clound.point[i].pixel_y = clound.indices[i] / width; where Point[i].pixel_x is the x coordinate, Point[i].pixel_y is the y coordinate, Indices[i] is the pixel index value, and Width is the image width;

[0072] Step 32: After the object point cloud is projected to the pixel coordinates, the 2D bounding box of the object point cloud in the pixel coordinates is obtained by comparing the maximum and minimum values ​​of the point cloud in the pixel coordinate axis direction;

[0073] Step 33: perform matching similarity calculation between the back-projected two-dimensional bounding box of the object point cloud and the circumscribed rectangular bounding box output by the target detection algorithm, record the object point cloud that is most similar to the object identified by the target detection, generate a labeled 3D target object virtual object model, and superimpose the virtual object model and the real environment, wherein the similarity calculation uses IOU, border center distance and size difference as calculation indicators for matching similarity.

[0074] like Figure 3 and Figure 4 As shown, the IOU is used to represent the overlap rate between the target area generated by the model and the original marked real area, that is, the intersection of the detection border and the real border is divided by the union of the detection border and the real border. The calculation formula is:

[0075]

[0076] like Figure 3 and Figure 4 As shown, the calculation method of the frame center distance includes the following steps:

[0077] The distance between the centers of the two borders P1 and P2 is used to describe the distance similarity, and the size difference is used to describe the size similarity. size match By using IOU, border center distance and size difference as calculation indicators of matching similarity in the similarity calculation, the matching similarity between the object point cloud projection frame and the target detection result bounding box can be described, and the point cloud segmentation result and the target detection result can be fused, where x represents the horizontal coordinate and vertical coordinate of point p.

[0078] like Figure 5 and Figure 8 As shown, the superposition of the virtual object model and the real environment, the superposition method includes the following steps:

[0079] Step 331: extract the projected quadrilateral region and the feature points in the quadrilateral region from the video stream, adjust the size and position of the tracking window according to the current feature point position and movement direction, if the marker is visible, proceed to step 332; if the marker is not visible, but more than four non-collinear feature points in the area to be registered are tracked successfully, proceed to step 333, otherwise set the three-dimensional transformation matrix of the current frame to be the same as the previous frame, and proceed to step 334;

[0080] Step 332: The entire logo is visible, and the three-dimensional transformation matrix and the camera intrinsic parameter matrix are directly calculated based on the logo projection, and then the process goes to step 334;

[0081] Step 333: Calculate the current 3D transformation matrix using the homography relationship between planes and the transformation matrix of the previous frame. If the internal parameters of the camera do not change, proceed to step 334.

[0082] Step 334: Use the transformation matrix to achieve three-dimensional registration and go to step 331.

[0083] In the wartime field intelligent rescue system described in the embodiment of the present invention, in order to achieve the effect of superimposing and stabilizing the virtual object model and the real environment, multiple cameras are used to scan the physical space in real time, and real-time calculations are performed based on the image data collected by the cameras to achieve reconstruction of the three-dimensional scene. The main steps are:

[0084] (1) Camera calibration. Since the camera is distorted when it leaves the factory, in order to obtain accurate data, it is necessary to perform camera calibration based on the various postures of the chessboard before use.

[0085] (2) Image alignment: Since the two cameras are located at different positions and the scenes they see are different, it is necessary to match the left and right images of the same scene to form a depth map.

[0086] (3) Three-dimensional rendering: The actual scene data is acquired through camera acquisition. Based on the camera posture, the position and posture changes of the current camera and the initial camera are tracked through an iterative alignment algorithm. The depth data with known posture is fused into the three-dimensional space. The feature tracking method and the spatial positioning method are combined to achieve accurate fusion between the real scene and the virtual object. The fiducial marker method is simple to implement, but errors will still occur when multiple fiducial markers are occluded. Relatively speaking, the feature method can more effectively solve the occlusion problem.

[0087] like Figure 6 As shown, the superposition of the virtual object model and the real environment, the superposition method also includes the following steps:

[0088] Step 3311: Ensure that each feature point is located at the intersection of multiple straight lines;

[0089] Step 3312: adopting a virtual-real registration method based on a global homography matrix, using the global homography matrix corresponding to the previous frame to transform the initial image, and performing feature matching between the transformed image and the current image;

[0090] Step 3313: Establish a homography relationship between the current frame and the initial frame.

[0091] In the wartime field intelligent rescue system described in the embodiment of the present invention, by adopting the virtual-real registration method based on the global homography matrix, the initial image is transformed using the global homography matrix corresponding to the previous frame, and the transformed image is feature matched with the current image. This ensures the effectiveness of the algorithm, while avoiding the problem of error accumulation, effectively improving the correctness of feature point matching, and maximizing the robustness of the algorithm.

[0092] like Figure 6 As shown, the superposition of the virtual object model and the real environment, the superposition method also includes the following steps:

[0093] Step 33121: extracting image feature points;

[0094] Step 33122: achieving effective matching between image feature points and local map points through image feature descriptors;

[0095] Step 33123: Calculate the map point reprojection error, use the bundle adjustment method to optimize the camera posture, and realize the tracking and positioning of the camera at the moment of each frame image acquisition. If the tracking and positioning fails, use the repositioning method based on the bag-of-words model to find the key frame that is closest to the current frame description, and match the key frame with the current frame for feature points and posture estimation.

[0096] In the wartime field intelligent rescue system described in the embodiment of the present invention, the ORB-SLAM2 method stores scene information such as map points and key frames in the map, and then performs global pose optimization based on the map element optimization. The algorithm map is composed of key frames, map points, co-viewing images and spanning trees. The key frame can record most of the information of the adjacent frames. Only when the information transformation of the feature points of the frame image is greater than the threshold, the current frame will be recorded as a key frame. The map point is composed of the three-dimensional world coordinate points of the feature points extracted from the image. The co-viewing image is used to record the relationship between different key frames, which can describe how many of the same map points can be seen in different key frames. In the co-viewing image, each key frame is a node. If the number of co-viewing map points between two key frames is greater than the threshold value, an edge is established between the two nodes, and the weight of the edge is the number of map points visible to the two key frames. The spanning tree is a subset of the co-viewing image, which retains all nodes (i.e., key frames) of the co-viewing image, but each node only retains the edge between the key frames with the most co-viewing map points.

[0097] In the wartime field intelligent rescue system described in the embodiment of the present invention, the main function of the algorithm tracking thread is to estimate the camera pose of each frame and determine whether the frame is a key frame. Its input is the camera acquisition image of each frame. The thread first extracts the image feature points, and then uses the feature descriptor to achieve effective matching of the image feature points and local map points. Then the map point reprojection error is calculated, and the camera pose is optimized using the bundle adjustment method to achieve camera tracking and positioning at the moment of each frame image acquisition. If tracking and positioning fails, the key frame that is closest to the current frame description is found using the repositioning method based on the bag of words model, and the key frame is matched with the current frame for feature point matching and pose estimation.

[0098] In the wartime field intelligent rescue system described in the embodiment of the present invention, the sensor data image frame of the sparse point cloud occlusion algorithm based on the semantic depth map contains various types of object information, so the semantic recognition advantage of deep learning combined with the position information provided by the virtual and real occlusion algorithm can extract the semantic information of the scene and establish a scene semantic map. Considering the portability requirements of augmented reality devices, it is difficult for augmented reality devices to carry high-performance GPUs, and point cloud semantic segmentation or image semantic segmentation SegNet, MASK-RCNN and other methods in deep learning take a long time without GPU acceleration, and cannot guarantee the real-time requirements of the algorithm. The present invention ensures the real-time performance of the algorithm through step 33123, but the image target detection method can only obtain two-dimensional image semantic information, and loses depth information. The three-dimensional data is projected into a two-dimensional space through point cloud object segmentation, which reduces the semantic error caused by the output information of the target detection algorithm as a plane rectangular bounding box. Through semantic information, dynamic objects in the scene are identified and feature points on dynamic objects are extracted.

[0099] like Figure 7 As shown in the figure, after the semantic database is updated, the system records the local semantic point cloud obtained. Due to the limited resources of augmented reality wearable devices, it is costly to save all the dense point clouds of the scene. The application is only concerned with the semantic point cloud information. For other dense point cloud information of the scene, the application actually only needs to obtain its surface conditions to facilitate the interaction between the application and the environment. Only the dense point cloud with local semantics is retained, and the greedy triangulation meshing method is adopted for the scene background to generate the global massive dense point cloud data into a single mesh.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for the technical features of the components or all of the components therein. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A wartime field intelligent rescue system, characterized in that: The system comprises: The wartime module is used for soldiers to identify and search for combat injury information in the multimodal combat injury database through keyword text, image and / or voice, so as to retrieve and match combat injury information and obtain accurate treatment plans based on the retrieved and matched information; If it is during combat, the collected image, voice, and text data will be automatically saved in the multimodal battle injury database, and after the data collected by the smart interactive glasses is recovered to the military headquarters, the data will be transmitted to the multimodal battle injury database for intelligent situational image analysis; A training module, wherein the training module includes a training parameter setting, which is used to autonomously select training parameters, including combat environment and combat injury categories; a virtual combat injury treatment, which is used to treat combat injuries of virtual people in the training parameters; The intelligent scene image analysis includes the following steps: Step 1: Find a point P in the space, find the n points closest to point P through KdTree, calculate the distance from n points to point P, and classify the points whose distance is less than the threshold into the Q category; Step 2: Find the unsearched points in class Q and repeat step 1; Step 3: The search is completed until all points in the Q class are searched. After the plane point cloud in the scene is segmented, the object point clouds are extracted one by one. The scene objects are segmented by setting a threshold through the Euclidean clustering method, and objects of different point cloud scales in the scene are extracted; The intelligent scene image analysis also includes the following steps: Step 31: After the object point cloud is segmented, the object point cloud group is projected onto the 2D plane by calculation, and the pixel coordinates of each point are calculated as: clound.point[i].pixel_x=clound.indices[i]%width; clound.point[i].pixel_y=clound.indices[i] / width, where, clound.point[i].pixel_x is the horizontal coordinate, clound.point[i].pixel_y is the vertical coordinate, clound.indices[i] is the pixel index value, and width is the image width; Step 32: After the object point cloud is projected to the pixel coordinates, the 2D bounding box of the object point cloud in the pixel coordinates is obtained by comparing the maximum and minimum values ​​of the point cloud in the pixel coordinate axis direction; Step 33: perform matching similarity calculation between the back-projected two-dimensional bounding box of the object point cloud and the circumscribed rectangular bounding box output by the target detection algorithm, record the object point cloud that is most similar to the object identified by the target detection, generate a labeled 3D target object virtual object model, and superimpose the virtual object model and the real environment, wherein the similarity calculation uses IOU, border center distance and size difference as calculation indicators for matching similarity; The IOU is used to represent the overlap rate between the target area generated by the model and the original marked real area, that is, the intersection of the detection frame and the real frame is divided by the union of the detection frame and the real frame. The calculation formula is: The calculation method of the frame center distance includes the following steps: The distance between the centers of the two bounding boxes p1 and p2 is used to describe the distance similarity, and the size difference is used to describe the size similarity, where: match Among them, x represents the horizontal coordinate of point p and y represents the vertical coordinate of point p.

2. The wartime field intelligent rescue system according to claim 1, characterized in that: The virtual combat injury treatment further comprises the following steps: By setting the training parameters, the battle injury scene, battle injury category and location are virtually constructed on the virtual human; Soldiers perform simulated combat injury treatment operations on virtual humans according to the treatment tools provided by the training parameter settings, wherein the simulated combat injury treatment operations include scaling or rotating the virtual humans, controlling and accurately locating combat injury details and implementing treatment.

3. The wartime field intelligent rescue system according to claim 1, characterized in that: The superposition of the virtual object model and the real environment, the superposition method comprises the following steps: Step 331: extract the projected quadrilateral region and the feature points in the quadrilateral region from the video stream, adjust the size and position of the tracking window according to the current feature point position and movement direction, if the marker is visible, proceed to step 332; if the marker is not visible, but more than four non-collinear feature points in the area to be registered are tracked successfully, proceed to step 333, otherwise set the three-dimensional transformation matrix of the current frame to be the same as the previous frame, and proceed to step 334; Step 332: The entire logo is visible, and the three-dimensional transformation matrix and the camera intrinsic parameter matrix are directly calculated based on the logo projection, and then the process goes to step 334; Step 333: Calculate the current 3D transformation matrix using the homography relationship between planes and the transformation matrix of the previous frame. If the internal parameters of the camera do not change, proceed to step 334. Step 334: Use the transformation matrix to achieve three-dimensional registration and go to step 331.

4. The wartime field intelligent rescue system according to claim 3, characterized in that: The superposition of the virtual object model and the real environment, the superposition method also includes the following steps: Step 3311: Ensure that each feature point is located at the intersection of multiple straight lines; Step 3312: adopting a virtual-real registration method based on a global homography matrix, using the global homography matrix corresponding to the previous frame to transform the initial image, and performing feature matching between the transformed image and the current image; Step 3313: Establish a homography relationship between the current frame and the initial frame.

5. The wartime field intelligent rescue system according to claim 4, characterized in that: The superposition of the virtual object model and the real environment, the superposition method also includes the following steps: Step 33121: extracting image feature points; Step 33122: achieving effective matching between image feature points and local map points through image feature descriptors; Step 33123: Calculate the map point reprojection error, use the bundle adjustment method to optimize the camera posture, and achieve camera tracking and positioning at the moment of each frame image acquisition.

6. The wartime field intelligent rescue system according to claim 5, characterized in that: In the step 33123, if the tracking and positioning fails, a repositioning method is used based on a bag-of-words model to find a key frame that is closest to the description of the current frame, and feature point matching and pose estimation are performed on the key frame and the current frame.

Images