Multi-modal visual information processing method and device

Abstract

The invention discloses a multi-mode visual information processing method and device. The method comprises the following steps: acquiring visual information to be processed; the to-be-processed visual information comprises first to-be-processed visual information and second to-be-processed visual information; performing calculation processing on the visual information to be processed to obtain multi-model visual information; the multi-modal visual information comprises first modal visual information and second modal visual information; the first modal visual information comprises first modal coordinate information and first modal vector information; the first modal coordinate information comprises first sub-modal coordinate information and second sub-modal coordinate information; the second modal visual information comprises second modal coordinate information and second modal vector information; the second modal coordinate information comprises third sub-modal coordinate information and fourth sub-modal coordinate information; and determining target visual processing result information based on the to-be-processed visual information and the multi-model visual information.

Description

Technical Field

[0001] The present invention relates to the field of visual technology, and in particular to a multimodal visual information processing method and device. Background Art

[0002] At present, the solution commonly adopted by mainstream eye tracking companies is pupil corneal reflection technology based on infrared light assistance. This method uses infrared light spots to assist and is easy to extract pupil features in the eye for tracking. However, this method also has many defects. On the one hand, it relies on precise optical instruments and powerful detection algorithms. Once an error occurs, the accuracy of the algorithm will drop sharply. On the other hand, it is not robust enough to problems such as eye slippage. For example, for desktop eye trackers, it cannot cope with head rotation. When the head moves slightly out of the camera's shooting range, it is impossible to capture eye images stably. For head-mounted eye trackers, although head movement does not need to be considered, there will be problems with device slippage. Usually, calibration is required to determine the position of the eye tracking device relative to the human eye. However, once the eye tracker causes the device to slip due to factors such as movement, the relative position relationship originally saved in the algorithm will be affected, and the accuracy will decrease. At this time, the user needs to re-calibrate, which is time-consuming and labor-intensive, thereby affecting the user experience. Therefore, a multimodal visual information processing method and device are provided to extract high-quality eye features and improve the accuracy and robustness of eye tracking. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a multimodal visual information processing method and device that is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0004] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a multimodal visual information processing method, the method comprising:

[0005] Acquire visual information to be processed; the visual information to be processed includes first visual information to be processed and second visual information to be processed;

[0006] The visual information to be processed is processed to obtain multi-modal visual information; the multi-modal visual information includes first modal visual information and second modal visual information; the first modal visual information includes first modal coordinate information and first modal vector information; the first modal coordinate information includes first sub-modal coordinate information and second sub-model coordinate information; the second modal visual information includes second modal coordinate information and second modal vector information; the second modal coordinate information includes third sub-modal coordinate information and fourth sub-modal coordinate information;

[0007] Based on the visual information to be processed and the multi-model visual information, target visual processing result information is determined.

[0008] A second aspect of an embodiment of the present invention discloses a multimodal visual information processing device, the device comprising:

[0009] An acquisition module is used to acquire visual information to be processed; the visual information to be processed includes first visual information to be processed and second visual information to be processed;

[0010] A processing module, used for performing computational processing on visual information to be processed to obtain multi-modal visual information; the multi-modal visual information includes first modal visual information and second modal visual information; the first modal visual information includes first modal coordinate information and first modal vector information; the first modal coordinate information includes first sub-modal coordinate information and second sub-model coordinate information; the second modal visual information includes second modal coordinate information and second modal vector information; the second modal coordinate information includes third sub-modal coordinate information and fourth sub-modal coordinate information;

[0011] The determination module is used to determine target visual processing result information based on the visual information to be processed and the multi-model visual information.

[0012] The third aspect of the present invention discloses another multimodal visual information processing device, the device comprising:

[0013] A memory storing executable program code;

[0014] a processor coupled to the memory;

[0015] The processor calls the executable program code stored in the memory to execute part or all of the steps in the multimodal visual information processing method disclosed in the first aspect of the embodiment of the present invention.

[0016] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the multimodal visual information processing method disclosed in the first aspect of an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flowchart of a multimodal visual information processing method disclosed in an embodiment of the present invention;

[0019] Figure 2is a structural schematic diagram of a multimodal visual information processing device disclosed in an embodiment of the present invention;

[0020] Figure 3 It is a structural schematic diagram of another multimodal visual information processing device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0023] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0024] The present invention discloses a multimodal visual information processing method and device, which are conducive to high-quality eye feature extraction and improve eye tracking accuracy and robustness. The following are detailed descriptions.

[0025] Embodiment 1

[0026] See also Figure 1 , Figure 1 is a flow chart of a multimodal visual information processing method disclosed in an embodiment of the present invention. Figure 1 The multimodal visual information processing method described is applied to a management system, such as a local server or a cloud server for management, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the multimodal visual information processing method may include the following operations:

[0027] 101. Obtain visual information to be processed.

[0028] In an embodiment of the present invention, the visual information to be processed includes first visual information to be processed and second visual information to be processed.

[0029] 102. Perform computational processing on the visual information to be processed to obtain multi-model visual information.

[0030] In an embodiment of the present invention, the multimodal visual information includes first modal visual information and second modal visual information; the first modal visual information includes first modal coordinate information and first modal vector information; the first modal coordinate information includes first sub-modal coordinate information and second sub-model coordinate information; the second modal visual information includes second modal coordinate information and second modal vector information; the second modal coordinate information includes third sub-modal coordinate information and fourth sub-modal coordinate information.

[0031] 103. Based on the visual information to be processed and the multi-model visual information, target visual processing result information is determined.

[0032] It should be noted that the multimodal visual information processing method of the present application reduces the sensitivity to light and camera slippage by processing the data information of the microsensor corresponding to the second modal visual information, because the contact lens is always close to the eye. Even if the user is not facing the camera directly, the microsensor can still sense the movement of the eyeball. Furthermore, through the fusion of multimodal visual information, it is possible to adjust the spatial position information of the microsensor by fusing the image visual information, thereby improving the estimation accuracy of the three-dimensional optical axis vector.

[0033] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0034] In an optional embodiment, the visual information to be processed is subjected to computational processing to obtain multi-model visual information, including:

[0035] Performing computational processing on the first visual information to be processed to obtain first modal visual information;

[0036] The second visual information to be processed is computationally processed to obtain second modal visual information.

[0037] It should be noted that the first modality visual information represents the visual information of the image, which may be the visual information corresponding to a single-camera multi-light source system.

[0038] It should be noted that the second modality visual information represents the visual information of the micro sensor, which may be position information, and the embodiment of the present invention does not limit this.

[0039] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0040] In another optional embodiment, performing computational processing on the first visual information to be processed to obtain the first modal visual information includes:

[0041] Performing calculation processing on the first visual information to be processed to obtain first modal coordinate information;

[0042] The first modal coordinate information is vector constructed to obtain the first modal vector information; the vector direction of the first modal vector information points from the coordinate point corresponding to the first sub-modal coordinate information to the coordinate point corresponding to the second sub-modal coordinate information.

[0043] It should be noted that the above-mentioned first modal vector information represents the three-dimensional optical axis vector of the eyeball in the first modality.

[0044] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0045] In yet another optional embodiment, the first visual information to be processed is calculated to obtain the first modal coordinate information, including:

[0046] Using the first modal calculation model to calculate and process the first visual information to be processed to obtain first sub-modal coordinate information;

[0047] Among them, the first modal calculation model is:

[0048]

[0049] Where MTZB1 is the coordinate information of the first sub-modal; XZ1 is the coordinate information of the center point in the first visual information to be processed; GY 1 and GY 2 are respectively the first light source vector information in the first visual information to be processed and the second light source vector information in the first visual information to be processed; BD 1 and BD 2 are respectively the first light spot coordinate information in the first visual information to be processed and the second light spot coordinate information in the first visual information to be processed; XS1 is the first coefficient;

[0050] The first submodal coordinate information and the first visual information to be processed are solved to obtain the second submodal coordinate information.

[0051] It should be noted that the above-mentioned solution processing of the first submodal coordinate information and the first visual information to be processed can be based on a neural network model, can be based on a solution rule set by a user, or can be implemented based on other artificial intelligence algorithms, which is not limited in the embodiments of the present invention.

[0052] It should be noted that the first light spot coordinate information in the first visual information to be processed and the second light spot coordinate information in the first visual information to be processed represent the Purkinje spots imaged in the camera.

[0053] It should be noted that the center point coordinate information in the first visual information to be processed may be the point corresponding to the optical center of the camera.

[0054] It should be noted that the first coefficient may be set by a user or a default value given by the system, and the embodiment of the present invention does not limit this.

[0055] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0056] In yet another optional embodiment, performing computational processing on the second visual information to be processed to obtain the second modal visual information includes:

[0057] The second visual information to be processed is calculated using the second model calculation model to obtain third sub-modal coordinate information;

[0058] Among them, the second model calculation model is:

[0059]

[0060] Where MTZB2 is the coordinate information of the third sub-modal; AZB 1 AZB 2 AZB 3 and AZB 4 The first sensor coordinate point information, the second sensor coordinate point information, the third sensor coordinate point information and the fourth sensor coordinate point information in the second visual information to be processed are respectively;

[0061] The second visual information to be processed is calculated using the third model calculation model to obtain fourth sub-modal coordinate information;

[0062] Among them, the calculation model of the third model is:

[0063] MTZB4=(XS2*AZB 1 +XS3*AZB 2 +XS4*AZB 3 ) / 3;

[0064] Wherein, MTZB4 is the coordinate information of the fourth submodal; XS2, XS3 and XS4 are the second coefficient, the third coefficient and the fourth coefficient respectively;

[0065] The third submodal coordinate information and the fourth submodal coordinate information are vector constructed to obtain the second modal vector information; the vector direction of the second modal vector information points from the coordinate point corresponding to the third submodal coordinate information to the coordinate point corresponding to the fourth submodal coordinate information.

[0066] It should be noted that the coordinate points corresponding to the first sensor coordinate point information, the second sensor coordinate point information, the third sensor coordinate point information and the fourth sensor coordinate point information are not coplanar in the three-dimensional space, that is, the above four points cannot be contained in the same plane in the three-dimensional space.

[0067] It should be noted that the second coefficient, the third coefficient and the fourth coefficient may be set by a user or given by a system, and the embodiment of the present invention does not limit this.

[0068] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0069] In an optional embodiment, the above-mentioned determining the target visual processing result information based on the visual information to be processed and the multi-model visual information includes:

[0070] Calculate and process the visual information to be processed and the multi-model visual information to obtain the modal coordinate distance value;

[0071] Determine whether the modal distance value is greater than or equal to the distance threshold, and obtain a distance judgment result;

[0072] When the modal judgment result is yes, the third modal vector information is determined based on the modal coordinate distance value and the second modal vector information;

[0073] When the modal judgment result is no, determining the second modal vector information as the third modal vector information;

[0074] The first modal vector information and the third modal vector information are fused to obtain target visual processing result information.

[0075] It should be noted that the above distance threshold may be set by a user or given by a system, and the embodiment of the present invention does not limit this.

[0076] It should be noted that the above-mentioned fusion processing of the first modal vector information and the third modal vector information can average the data information obtained from the two modalities, or can be a selective fusion, that is, when the light brightness corresponding to the first visual information to be processed is greater than the preset brightness threshold, the third modal vector information is used, otherwise the first modal vector information is used, and the embodiment of the present invention is not limited. Furthermore, using a multimodal strategy, a modal data such as microsensor position information is added on the basis of image information as an auxiliary to the eye algorithm, thereby improving the overall performance of the algorithm. Even if the position information is not good, the algorithm can be degenerated into an algorithm that relies only on eye images, thereby improving the robustness of the model.

[0077] In this optional embodiment, as an optional implementation manner, the third modal vector information is determined based on the modal coordinate distance value and the second modal vector information, including:

[0078] Take one third of the modal coordinate distance value as the target distance value;

[0079] The target distance value is subtracted from the second modal vector information to obtain the third modal vector information.

[0080] It should be noted that the above-mentioned adjustment of the modal vector using the modal coordinate distance value is to adjust the three-dimensional optical axis vector estimated by the microsensor flow according to the slip distance in the image, so as to generate a corrected optical axis vector, that is, to fuse the information of the two modes of image and microsensor, and correct the position coordinates to ensure the reliability of the three-dimensional optical axis vector estimation.

[0081] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0082] In another optional embodiment, the visual information to be processed and the multi-model visual information are calculated and processed to obtain the modal coordinate distance value, including:

[0083] The fourth modality calculation model is used to calculate and process the visual information to be processed to obtain the fifth sub-modality coordinate information;

[0084] Among them, the fourth modal calculation model is:

[0085]

[0086] Where MTZB5 is the fifth submodal coordinate information; XS5 is the fifth coefficient;

[0087] A distance calculation is performed on the fifth sub-modal coordinate information and the second sub-modal coordinate information to obtain a modal coordinate distance value.

[0088] It should be noted that the fifth coefficient may be set by a user or given by a system, and the embodiment of the present invention does not limit this.

[0089] It can be seen that implementing the multimodal visual information processing method described in the embodiment of the present invention is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0090] Embodiment 2

[0091] See also Figure 2 , Figure 2 is a schematic diagram of the structure of a multimodal visual information processing device disclosed in an embodiment of the present invention. Figure 2 The described device can be applied to a warehouse management system, such as a local server or cloud server for warehouse logistics multimodal visual information processing management, etc., and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the device may include:

[0092] An acquisition module 201 is used to acquire visual information to be processed; the visual information to be processed includes first visual information to be processed and second visual information to be processed;

[0093] The processing module 202 is used to perform computational processing on the visual information to be processed to obtain multi-modal visual information; the multi-modal visual information includes first modal visual information and second modal visual information; the first modal visual information includes first modal coordinate information and first modal vector information; the first modal coordinate information includes first sub-modal coordinate information and second sub-model coordinate information; the second modal visual information includes second modal coordinate information and second modal vector information; the second modal coordinate information includes third sub-modal coordinate information and fourth sub-modal coordinate information;

[0094] The determination module 203 is used to determine target visual processing result information based on the visual information to be processed and the multi-model visual information.

[0095] It can be seen that implementation Figure 2 The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0096] In another optional embodiment, Figure 2 As shown, the processing module 202 performs computational processing on the visual information to be processed to obtain multi-model visual information, including:

[0097] Performing computational processing on the first visual information to be processed to obtain first modal visual information;

[0098] The second visual information to be processed is computationally processed to obtain second modal visual information.

[0099] It can be seen that implementation Figure 2 The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0100] In yet another optional embodiment, Figure 2 As shown, the processing module 202 performs computational processing on the first visual information to be processed to obtain the first modal visual information, including:

[0101] Performing calculation processing on the first visual information to be processed to obtain first modal coordinate information;

[0102] The first modal coordinate information is vector constructed to obtain the first modal vector information; the vector direction of the first modal vector information points from the coordinate point corresponding to the first sub-modal coordinate information to the coordinate point corresponding to the second sub-modal coordinate information.

[0103] It can be seen that implementation Figure 2 The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0104] In yet another optional embodiment, Figure 2 As shown, the processing module 202 performs calculation processing on the first visual information to be processed to obtain the first modal coordinate information, including:

[0105] Using the first modal calculation model to calculate and process the first visual information to be processed to obtain first sub-modal coordinate information;

[0106] Among them, the first modal calculation model is:

[0107]

[0108] Where MTZB1 is the coordinate information of the first sub-modal; XZ1 is the coordinate information of the center point in the first visual information to be processed; GY 1 and GY 2 are respectively the first light source vector information in the first visual information to be processed and the second light source vector information in the first visual information to be processed; BD 1 and BD 2 are respectively the first light spot coordinate information in the first visual information to be processed and the second light spot coordinate information in the first visual information to be processed; XS1 is the first coefficient;

[0109] The first submodal coordinate information and the first visual information to be processed are solved to obtain the second submodal coordinate information.

[0110] It can be seen that implementation Figure 2The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0111] In yet another optional embodiment, Figure 2 As shown, the processing module 202 performs computational processing on the second visual information to be processed to obtain the second modal visual information, including:

[0112] The second visual information to be processed is calculated using the second model calculation model to obtain third sub-modal coordinate information;

[0113] Among them, the second model calculation model is:

[0114]

[0115] Where MTZB2 is the coordinate information of the third sub-modal; AZB 1 AZB 2 AZB 3 and AZB 4 The first sensor coordinate point information, the second sensor coordinate point information, the third sensor coordinate point information and the fourth sensor coordinate point information in the second visual information to be processed are respectively;

[0116] The second visual information to be processed is calculated using the third model calculation model to obtain fourth sub-modal coordinate information;

[0117] Among them, the calculation model of the third model is:

[0118] MTZB4=(XS2*AZB 1 +XS3*AZB 2 +XS4*AZB 3 ) / 3;

[0119] Wherein, MTZB4 is the coordinate information of the fourth submodal; XS2, XS3 and XS4 are the second coefficient, the third coefficient and the fourth coefficient respectively;

[0120] The third submodal coordinate information and the fourth submodal coordinate information are vector constructed to obtain the second modal vector information; the vector direction of the second modal vector information points from the coordinate point corresponding to the third submodal coordinate information to the coordinate point corresponding to the fourth submodal coordinate information.

[0121] It can be seen that implementation Figure 2 The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0122] In yet another optional embodiment, Figure 2As shown, the determination module 203 determines the target visual processing result information based on the visual information to be processed and the multi-model visual information, including:

[0123] Calculate and process the visual information to be processed and the multi-model visual information to obtain the modal coordinate distance value;

[0124] Determine whether the modal distance value is greater than or equal to the distance threshold, and obtain a distance judgment result;

[0125] When the modal judgment result is yes, the third modal vector information is determined based on the modal coordinate distance value and the second modal vector information;

[0126] When the modal judgment result is no, determining the second modal vector information as the third modal vector information;

[0127] The first modal vector information and the third modal vector information are fused to obtain target visual processing result information.

[0128] It can be seen that the implementation Figure 2 The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0129] In yet another optional embodiment, Figure 2 As shown, the determination module 203 performs calculation processing on the visual information to be processed and the multi-model visual information to obtain the modal coordinate distance value, including:

[0130] The fourth modality calculation model is used to calculate and process the visual information to be processed to obtain the fifth sub-modality coordinate information;

[0131] Among them, the fourth modal calculation model is:

[0132]

[0133] Where MTZB5 is the fifth submodal coordinate information; XS5 is the fifth coefficient;

[0134] A distance calculation is performed on the fifth sub-modal coordinate information and the second sub-modal coordinate information to obtain a modal coordinate distance value.

[0135] It can be seen that the implementation Figure 2 The described multimodal visual information processing device is conducive to high-quality eye feature extraction and improves eye tracking accuracy and robustness.

[0136] Embodiment 3

[0137] See also Figure 3 , Figure 3 is a schematic diagram of the structure of another multimodal visual information processing device disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a warehouse management system, such as a local server or cloud server for warehouse logistics multimodal visual information processing management, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the device may include:

[0138] A memory 301 storing executable program codes;

[0139] a processor 302 coupled to the memory 301;

[0140] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the multimodal visual information processing method described in the first embodiment.

[0141] Embodiment 4

[0142] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the multimodal visual information processing method described in the first embodiment.

[0143] Embodiment 5

[0144] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the multimodal visual information processing method described in Embodiment 1.

[0145] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0146] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0147] Finally, it should be noted that the multimodal visual information processing method and device disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal visual information processing method, characterized in that: Methods include: Obtain visual information to be processed; The visual information to be processed includes first visual information to be processed and second visual information to be processed; The visual information to be processed is processed to obtain multi-modal visual information; the multi-modal visual information includes first modal visual information and second modal visual information; the first modal visual information includes first modal coordinate information and first modal vector information; the first modal coordinate information includes first sub-modal coordinate information and second sub-model coordinate information; the second modal visual information includes second modal coordinate information and second modal vector information; the second modal coordinate information includes third sub-modal coordinate information and fourth sub-modal coordinate information; Based on the visual information to be processed and the multi-model visual information, target visual processing result information is determined.

2. The multimodal visual information processing method according to claim 1, characterized in that: The visual information to be processed is processed to obtain multi-model visual information, including: Performing computational processing on the first visual information to be processed to obtain first modal visual information; The second visual information to be processed is computationally processed to obtain second modal visual information.

3. The multimodal visual information processing method according to claim 2, characterized in that: The first visual information to be processed is processed by calculation to obtain the first modal visual information, including: Performing calculation processing on the first visual information to be processed to obtain first modal coordinate information; The first modal coordinate information is vector constructed to obtain the first modal vector information; the vector direction of the first modal vector information points from the coordinate point corresponding to the first sub-modal coordinate information to the coordinate point corresponding to the second sub-modal coordinate information.

4. The multimodal visual information processing method according to claim 3, characterized in that: The first visual information to be processed is calculated to obtain first modal coordinate information, including: Using the first modal calculation model to calculate and process the first visual information to be processed to obtain first sub-modal coordinate information; Among them, the first modal calculation model is: Wherein, MTZB1 is the first submodal coordinate information; XZ1 is the center point coordinate information in the first visual information to be processed; GY1 and GY2 are the first light source vector information in the first visual information to be processed and the second light source vector information in the first visual information to be processed, respectively; BD1 and BD2 are the first light point coordinate information in the first visual information to be processed and the second light point coordinate information in the first visual information to be processed, respectively; XS1 is the first coefficient; The first submodal coordinate information and the first visual information to be processed are solved to obtain the second submodal coordinate information.

5. The multimodal visual information processing method according to claim 2, characterized in that: The second visual information to be processed is processed by calculation to obtain the second modal visual information, including: The second visual information to be processed is calculated using the second model calculation model to obtain third sub-modal coordinate information; Among them, the second model calculation model is: Wherein, MTZB2 is the third submodal coordinate information; AZB1, AZB2, AZB3 and AZB4 are the first sensor coordinate point information, the second sensor coordinate point information, the third sensor coordinate point information and the fourth sensor coordinate point information in the second visual information to be processed respectively; The second visual information to be processed is calculated using the third model calculation model to obtain fourth sub-modal coordinate information; Among them, the calculation model of the third model is: MTZB4=(XS2*AZB1+XS3*AZB2+XS4*AZB3) / 3; Wherein, MTZB4 is the coordinate information of the fourth submodal; XS2, XS3 and XS4 are the second coefficient, the third coefficient and the fourth coefficient respectively; The third submodal coordinate information and the fourth submodal coordinate information are vector constructed to obtain the second modal vector information; the vector direction of the second modal vector information points from the coordinate point corresponding to the third submodal coordinate information to the coordinate point corresponding to the fourth submodal coordinate information.

6. The multimodal visual information processing method according to claim 1, characterized in that: Based on the visual information to be processed and the multi-model visual information, the target visual processing result information is determined, including: Calculate and process the visual information to be processed and the multi-model visual information to obtain the modal coordinate distance value; Determine whether the modal distance value is greater than or equal to the distance threshold, and obtain a distance judgment result; When the modal judgment result is yes, the third modal vector information is determined based on the modal coordinate distance value and the second modal vector information; When the modal judgment result is no, determining the second modal vector information as the third modal vector information; The first modal vector information and the third modal vector information are fused to obtain target visual processing result information.

7. The multimodal visual information processing method according to claim 6, characterized in that: The visual information to be processed and the multi-model visual information are calculated and processed to obtain the modal coordinate distance value, including: The fourth modality calculation model is used to calculate and process the visual information to be processed to obtain the fifth sub-modality coordinate information; Among them, the fourth modal calculation model is: Where MTZB5 is the fifth submodal coordinate information; XS5 is the fifth coefficient; A distance calculation is performed on the fifth sub-modal coordinate information and the second sub-modal coordinate information to obtain a modal coordinate distance value.

8. A multimodal visual information processing device, characterized in that: The device includes: An acquisition module, used for acquiring visual information to be processed; The visual information to be processed includes first visual information to be processed and second visual information to be processed; A processing module, used for performing computational processing on visual information to be processed to obtain multi-modal visual information; the multi-modal visual information includes first modal visual information and second modal visual information; the first modal visual information includes first modal coordinate information and first modal vector information; the first modal coordinate information includes first sub-modal coordinate information and second sub-model coordinate information; the second modal visual information includes second modal coordinate information and second modal vector information; the second modal coordinate information includes third sub-modal coordinate information and fourth sub-modal coordinate information; The determination module is used to determine target visual processing result information based on the visual information to be processed and the multi-model visual information.

9. A multimodal visual information processing device, characterized in that: The device includes: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multimodal visual information processing method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the multimodal visual information processing method as claimed in any one of claims 1 to 7.

Images