A mobile multi-modal information acquisition terminal and an acquisition method thereof

Abstract

The application discloses a mobile multi-modal information acquisition terminal and a control method thereof, and belongs to the technical field of public safety and biological feature recognition. The terminal adopts a hardware and software integrated design. The hardware takes a central processing unit as a core, and is connected with a biological recognition module integrating a fingerprint, an iris, an ID card reading and a voiceprint acquisition through a bus, and an interactive display and a power management module. The software is based on an Android system to build a three-layer architecture of an application layer, a plug-in layer and a native driver layer. The terminal realizes one-stop acquisition, associated storage and efficient data interaction of multi-modal biological features through intelligent power consumption control of a page life cycle, a case-person core data association model and a built-in lightweight network service. The application improves the portability and acquisition efficiency of the equipment, guarantees the acquisition data quality, reduces the system maintenance cost, and is suitable for mobile law enforcement, outdoor investigation and other on-site acquisition scenes.

Description

Technical Field

[0001] This invention relates to the field of public safety and biometric identification technology, specifically to a mobile multimodal information collection terminal and its collection method. Background Technology

[0002] In public safety, criminal investigation, and population management, the collection of basic information and biometric information of individuals is a crucial step.

[0003] Existing data collection methods typically suffer from the following problems:

[0004] 1. The equipment is scattered and inconvenient to carry: Traditional data collection often requires carrying multiple sets of independent equipment (such as fingerprint scanners, iris scanners, ID card readers, voice recorders, etc.), which are not only heavy, but also cumbersome to deploy on site, making it difficult to meet the portability requirements of mobile law enforcement or outdoor investigation.

[0005] 2. Data fragmentation and integration difficulties: Different devices are usually manufactured by different companies, resulting in inconsistent data formats and a lack of a unified correlation mechanism. After collection, manual data aggregation and matching are often required, which can easily lead to information errors, omissions, or discrepancies between the data collected and the actual data.

[0006] 3. Complex operation and low efficiency: The operating procedures for each device vary, requiring a high level of technical expertise from the data acquisition personnel. In emergency or high-pressure environments, complex switching operations can easily lead to data acquisition failure or substandard data quality.

[0007] 4. Insufficient battery life and stability: When multiple devices work simultaneously, power management becomes chaotic, which can easily cause the mobile terminal's battery to be quickly depleted, affecting the ability to continue working in the field.

[0008] Therefore, there is an urgent need for a mobile multimodal information collection terminal that can highly integrate multiple biometric modules, unify operating procedures, and possess efficient data management capabilities. Summary of the Invention

[0009] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0010] Therefore, the purpose of this invention is to provide a mobile multimodal information acquisition terminal and its acquisition method to solve the problems mentioned in the background art.

[0011] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0012] A mobile multimodal information acquisition terminal includes a central processing unit, which is connected to a biometric module, an interaction and display module, and a power management module via a bus.

[0013] The terminal is equipped with a layered architecture software system based on the Android system. The software system works with the central processing unit, biometric module, interaction and display module and power management module to realize one-stop collection, associated storage and data interaction of multimodal biometric features.

[0014] The biometric module includes a fingerprint acquisition module, an iris acquisition module, an ID card reading module, and a voiceprint acquisition module.

[0015] The power management module is used to independently control the power-on and power-off of each peripheral module, thereby achieving intelligent regulation of the overall power consumption.

[0016] In a preferred embodiment of the mobile multimodal information acquisition terminal described in this invention, the fingerprint acquisition module is connected to the central processing unit via a USB interface and integrates an optical fingerprint sensor for acquiring rolling fingerprints and planar fingerprints.

[0017] The iris acquisition module is connected to an infrared camera via a USB interface. The infrared camera is used to transmit images via USB video protocol to capture images of both eyes' irises.

[0018] The ID card reading module integrates a radio frequency identification antenna and a decoding chip for reading resident ID card chip information;

[0019] The voiceprint acquisition module includes a built-in microphone array or an external audio input interface, and works with an audio processing unit to complete voiceprint recording.

[0020] The interaction and display module is a touch screen, used for human-computer interaction and real-time image preview during the acquisition process.

[0021] As a preferred embodiment of the mobile multimodal information acquisition terminal described in this invention, the layered architecture software system includes an application layer, a plug-in layer, and a native driver layer.

[0022] The application layer provides a unified business operation interface and is responsible for the flow of business logic, but does not directly operate the hardware.

[0023] The plugin layer encapsulates native driver capabilities into standard interfaces, providing a unified asynchronous call for the application layer;

[0024] The native driver layer communicates directly with the underlying hardware to complete hardware permission request processing, biometric algorithm library calls, and multi-threaded data processing operations.

[0025] As a preferred embodiment of the mobile multimodal information collection terminal described in this invention, the terminal is configured with an SQLite database, and the software system establishes a data model centered on cases and personnel based on the database. The collected biometric data are associated and bound to specific personnel files through a unique CollectionId.

[0026] The software system has a built-in lightweight web server and WebSocket service, which are used by external devices to synchronize data with the terminal via a local area network, push collection progress, and perform data export operations.

[0027] As a preferred embodiment of the mobile multimodal information acquisition terminal described in this invention, the power management module controls power consumption based on the lifecycle of the terminal acquisition page. When entering the acquisition page of the target biometric feature, the corresponding biometric module is automatically powered on, and when exiting the acquisition page, the corresponding biometric module is automatically powered off.

[0028] A method for acquiring multimodal information using a mobile multimodal information acquisition terminal, comprising the following steps:

[0029] S1. Establish a core data model of cases and personnel based on the SQLite database of the terminal, create personnel files and assign them unique collection identifiers;

[0030] S2. Based on the data collection requirements, power on the corresponding biometric module through the power management module to start the target biometric data collection process.

[0031] S3. Perform quality inspection and feature extraction on the collected raw biometric data, and bind and store the qualified feature data with the unique collection identifier of the personnel file.

[0032] S4. Perform local data management on the multimodal acquisition information within the terminal, or achieve data interaction with external devices through the terminal's network services.

[0033] As a preferred embodiment of the multimodal information collection method of the mobile multimodal information collection terminal described in this invention, in step S1, the resident ID card chip information is read through the ID card reading module, the terminal automatically parses and fills the personnel information form, generates a personnel file based on the form and stores it in the SQLite database, and assigns a unique CollectionId as a collection identifier to the personnel file.

[0034] In step S2, the power management module completes the automatic power-on and power-off control of the corresponding biometric module based on the lifecycle of the terminal acquisition page.

[0035] As a preferred embodiment of the multimodal information acquisition method of the mobile multimodal information acquisition terminal described in this invention, in step S3, the native driver layer calls the biometric algorithm library to perform quality assessment on the raw data, and only performs feature extraction and format conversion on the data that meets the data entry standards.

[0036] Among them, fingerprint collection involves acquiring images and compressing and extracting them after initializing the device; iris collection involves automatically taking pictures and cropping them to generate standard images after passing the algorithm detection; and voiceprint collection involves recording with noise reduction and encoding the original PCM data into a standard audio format. The storage path of each feature data is bound to the unique collection identifier in the personnel file and recorded in the database.

[0037] As a preferred embodiment of the multimodal information acquisition method of the mobile multimodal information acquisition terminal described in this invention, in step S4, the local data management method is as follows: structured data such as personnel attributes and acquisition time are directly stored in the SQLite database, and unstructured data such as fingerprint images and audio files are stored in the terminal application sandbox directory and the storage path reference is saved to the database.

[0038] The data writing process employs a transaction protection mechanism, merging multiple SQL insert statements for execution. If any statement fails, the entire process is rolled back. Furthermore, the application automatically performs database version checks and upgrades upon startup to ensure compatibility with older data.

[0039] As a preferred embodiment of the multimodal information collection method of the mobile multimodal information collection terminal described in this invention, in step S4, the data interaction with external devices includes: the terminal reads local cached data and database fields, and automatically renders and generates a collection report in docx format according to the public security standard typesetting format. The report supports direct printing or export to external storage media.

[0040] The terminal starts a background web server, and external devices access the terminal's IP address via the local area network to push the data collection progress in real time. The terminal also provides a packaged download interface to encapsulate the collected data of a specified case into a ZIP format for external devices to download and import into the backend platform.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] 1. This invention integrates multiple data collection functions, such as fingerprint collection, iris recognition, ID card information reading, and voiceprint collection, into a single handheld terminal device. This replaces the need to carry multiple separate devices in traditional operations, fundamentally reducing the equipment burden on data collection personnel. It enables a single person to complete the entire process with one hand, adapting to the portability needs of various scenarios such as mobile law enforcement and outdoor investigation. At the same time, it eliminates the cumbersome steps of deploying and debugging multiple devices on-site, effectively shortening the preparation time for on-site information collection and improving the on-site operational efficiency of public security, criminal investigation, and other work.

[0043] 2. A unified bus connecting all functional modules via a central processing unit enables integrated hardware control. Based on the Android system, a three-layer architecture (application layer, plugin layer, and native driver layer) is built, providing a unified business operation interface. This ensures that various biometric data collection operations follow the same workflow logic, significantly reducing the technical requirements for data collection personnel and avoiding collection errors caused by inconsistent operating procedures across multiple devices. Even in urgent, high-pressure environments, smooth data collection operations are guaranteed. Furthermore, this hybrid development architecture decouples business interface updates from underlying hardware drivers. Interface adjustments do not require recompiling the underlying drivers, and hardware module upgrades only require updating the corresponding plugins for adaptation. This significantly reduces the later maintenance costs and upgrade difficulty of terminal devices, enhancing the product's lifecycle value.

[0044] 3. This invention integrates a professional biometric algorithm library, which can perform real-time quality detection and feature extraction on the collected raw data such as fingerprints, irises, and voiceprints. Only data that meets the data entry standards is subsequently stored, thus eliminating the generation of low-quality and invalid data from the source of collection. At the same time, by establishing a data model centered on cases and personnel, all collected biometric data are automatically associated with specific personnel files through a unique collection identifier, realizing integrated and associated storage of multimodal data. This completely solves the problems of inconsistent data formats, errors and omissions in manual summarization, and discrepancies between personnel and files caused by traditional multi-device collection, ensuring the accuracy, standardization, and relevance of the collected information. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0046] Figure 1 This is a flowchart of a multimodal information acquisition method for a mobile multimodal information acquisition terminal according to the present invention. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] Example 1

[0049] The mobile multimodal information acquisition terminal disclosed in this embodiment is a handheld portable smart device. It adopts a central processing unit (CPU) core bus-type hardware architecture and is equipped with a three-layered software architecture based on the Android system. The hardware and software work together to realize intelligent and integrated control of multimodal information acquisition. The specific architecture is as follows:

[0050] Hardware architecture:

[0051] The terminal is mainly composed of a central processing unit (CPU) at its core, which is connected to the following modules via a bus:

[0052] 1. Biometric Module: Includes: 1) Fingerprint Acquisition Module: Connects via USB interface, integrates an optical fingerprint sensor, supports rolling fingerprint and planar fingerprint acquisition; 2) Iris Acquisition Module: Connects via USB interface to an infrared camera, supports USB Video Class (UVC) protocol image transmission, used for capturing iris images of both eyes; 3) ID Card Reading Module: Integrates RFID antenna and decoding chip, used for reading resident ID card chip information; 4) Voiceprint Acquisition Module: Utilizes built-in microphone array or external audio input, in conjunction with an audio processing unit, to record voiceprints;

[0053] 2. Interaction and Display Module: Touch screen for human-computer interaction and real-time image preview.

[0054] 3. Power Management Module: Independently controls the power-on and power-off of each peripheral module, reducing the overall power consumption of the device.

[0055] Software architecture:

[0056] The software platform runs on the Android system and adopts a layered architecture design: the software system is divided into three layers: the application layer (UniApp), the plugin layer (UniModule), and the native driver layer (Android Native).

[0057] 1. Application Layer: Provides a unified business operation interface. This includes functions such as case management, personnel information entry, biometric data collection guidance, and data query and export. This layer is responsible for business logic flow and does not directly operate the hardware.

[0058] 2. Plug-in layer: Acting as an intermediary bridge, it encapsulates native driver capabilities into standard JavaScript interfaces. Examples include FingerModule, IrisModule, and IDCardModule, enabling the application layer to control hardware through a unified asynchronous calling method.

[0059] 3. Native Driver Layer: Communicates directly with the underlying hardware, handling USB permission requests (such as UsbPermissionManager); calls various biometric algorithm libraries (SO libraries) for feature extraction, quality assessment, and format conversion (such as WLT decoding and fingerprint image segmentation); and implements complex multi-threaded tasks, such as real-time preview stream processing for iris detection and PCM to WAV audio encoding.

[0060] Example 2

[0061] Based on the mobile multimodal information collection terminal of Embodiment 1, this embodiment discloses a one-stop multimodal biometric data collection process for a single data collection object. This process uses the reading of resident ID card information as the basis for creating a personnel file, sequentially completing the collection of fingerprint, iris, and voiceprint biometric features. All collected data is associated with the personnel file through a unique CollectionId. Please refer to [link to relevant documentation]. Figure 1 The specific implementation steps are as follows:

[0062] 1. Case and Personnel Creation:

[0063] On the main interface, users click on Personnel Information Management. The system reads the second-generation ID card chip through the IDCardModule, automatically parses information such as name, ID number, and photo, fills in the form, generates personnel files, and stores them in the SQLite database.

[0064] 2. Fingerprint Acquisition: When a user enters the fingerprint acquisition interface, the application layer calls FingerModule.initDevice(); the native layer requests USB permission, and initializes the TrustFinger device upon successful request; when the user presses the fingerprint scanner, the native layer sends the fingerprint image data back to the application layer's Canvas for preview in real time; after acquisition, the native layer compresses and extracts features from the image, returns the file path, and the application layer binds it to the user ID.

[0065] 3. Iris Acquisition: Enter the iris acquisition interface, call IrisModule, start the UVC camera, and the native layer uses the IrisDetector algorithm to detect the position and clarity of the human eyes in the image in real time. When a qualified binocular iris image is detected, it automatically triggers the taking of a picture and crops and generates a standard iris image.

[0066] 4. Voiceprint Acquisition: Calls RecordingService to support noise-reduced recording. The waveform is displayed in real time during the recording process, and the raw PCM data is automatically encoded into WAV or MP3 format for storage after the recording ends.

[0067] Example 3

[0068] Based on the terminal architecture of Embodiment 1 and the data collection process of Embodiment 2, this embodiment discloses specific implementation methods for local storage and protection of collected data, standardized report generation, and local area network service-based export, achieving secure management and efficient flow of collected data, and adapting to the data analysis and backend platform storage requirements in the public safety field. The specific implementation is as follows:

[0069] 1. Local Storage: All structured data (personnel attributes, collection time, collection status) is stored in a local SQLite database on the terminal. Unstructured data (images, audio files) is stored in the application sandbox directory, with path references stored in the database.

[0070] 2. Database Version Control and Transaction Protection: Define CURRENT_DB_VERSION to automatically check and execute database upgrade scripts when the application starts, ensuring compatibility with old data. When writing data, use the executeTransaction method to merge multiple SQL insert statements for execution. For example, when saving a person's complete file, both the Person and BiometricCollection tables must be written simultaneously. If any step fails, the entire process is rolled back to prevent the generation of dirty data.

[0071] 3. Document Generation: The system uses the generate-docx.js tool to read locally cached images and database fields, and automatically renders and generates a collection report in docx format according to the public security standard layout format. It supports direct printing or export to external storage media.

[0072] 4. Service-based Export: The terminal starts a background service, FileServer (based on NanoHTTPD). Once the terminal is connected to the network, administrators can access the terminal's IP address via a PC browser. The terminal pushes real-time data collection progress to the PC via WebSocket and provides a packaged download interface to package all biometric data for a specified case into a ZIP file for download and import into the backend platform.

[0073] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A mobile multimodal information acquisition terminal, characterized in that, It includes a central processing unit, which is connected via a bus to a biometric module, an interaction and display module, and a power management module; The terminal is equipped with a layered architecture software system based on the Android system. The software system works with the central processing unit, biometric module, interaction and display module and power management module to realize one-stop collection, associated storage and data interaction of multimodal biometric features. The biometric module includes a fingerprint acquisition module, an iris acquisition module, an ID card reading module, and a voiceprint acquisition module. The power management module is used to independently control the power-on and power-off of each peripheral module, thereby achieving intelligent regulation of the overall power consumption.

2. The mobile multimodal information acquisition terminal according to claim 1, characterized in that, The fingerprint acquisition module is connected to the central processing unit via a USB interface and integrates an optical fingerprint sensor for acquiring both rolling and planar fingerprints. The iris acquisition module is connected to an infrared camera via a USB interface. The infrared camera is used to transmit images via USB video protocol to capture images of both eyes' irises. The ID card reading module integrates a radio frequency identification antenna and a decoding chip for reading resident ID card chip information; The voiceprint acquisition module includes a built-in microphone array or an external audio input interface, and works with an audio processing unit to complete voiceprint recording. The interaction and display module is a touch screen, used for human-computer interaction and real-time image preview during the acquisition process.

3. The mobile multimodal information acquisition terminal according to claim 1, characterized in that, The layered architecture software system includes an application layer, a plugin layer, and a native driver layer; The application layer provides a unified business operation interface and is responsible for the flow of business logic, but does not directly operate the hardware. The plugin layer encapsulates native driver capabilities into standard interfaces, providing a unified asynchronous call for the application layer; The native driver layer communicates directly with the underlying hardware to complete hardware permission request processing, biometric algorithm library calls, and multi-threaded data processing operations.

4. A mobile multimodal information acquisition terminal according to claim 1, characterized in that, The terminal is configured with an SQLite database, and the software system establishes a data model centered on cases and personnel based on the database. The collected biometric data are associated and bound to specific personnel files through a unique CollectionId. The software system has a built-in lightweight web server and WebSocket service, which are used by external devices to synchronize data with the terminal via a local area network, push collection progress, and perform data export operations.

5. A mobile multimodal information acquisition terminal according to claim 1, characterized in that, The power management module controls power consumption based on the lifecycle of the terminal's biometric data acquisition page. When entering the target biometric data acquisition page, the corresponding biometric module is automatically powered on, and when exiting the acquisition page, the corresponding biometric module is automatically powered off.

6. A multimodal information acquisition method based on the mobile multimodal information acquisition terminal according to any one of claims 1-5, characterized in that, The steps are as follows: S1. Establish a core data model of cases and personnel based on the SQLite database of the terminal, create personnel files and assign them unique collection identifiers; S2. Based on the data collection requirements, power on the corresponding biometric module through the power management module to start the target biometric data collection process. S3. Perform quality inspection and feature extraction on the collected raw biometric data, and bind and store the qualified feature data with the unique collection identifier of the personnel file. S4. Perform local data management on the multimodal acquisition information within the terminal, or achieve data interaction with external devices through the terminal's network services.

7. The mobile multimodal information acquisition method according to claim 6, characterized in that, In step S1, the resident ID card chip information is read through the ID card reading module, the terminal automatically parses and fills the personnel information form, generates a personnel file based on the form and stores it in the SQLite database, and assigns a unique CollectionId as a collection identifier to the personnel file. In step S2, the power management module completes the automatic power-on and power-off control of the corresponding biometric module based on the lifecycle of the terminal acquisition page.

8. The mobile multimodal information acquisition method according to claim 6, characterized in that, In step S3, the native driver layer calls the biometric algorithm library to perform quality assessment on the raw data, and only performs feature extraction and format conversion on the data that meets the data entry criteria. Among them, fingerprint collection involves acquiring images and compressing and extracting them after initializing the device; iris collection involves automatically taking pictures and cropping them to generate standard images after passing the algorithm detection; and voiceprint collection involves recording with noise reduction and encoding the original PCM data into a standard audio format. The storage path of each feature data is bound to the unique collection identifier in the personnel file and recorded in the database.

9. A mobile multimodal information acquisition method according to claim 6, characterized in that, In step S4, the local data management method is as follows: structured data such as personnel attributes and collection time are directly stored in the SQLite database, and unstructured data such as fingerprint images and audio files are stored in the terminal application sandbox directory and the storage path reference is saved to the database. The data writing process employs a transaction protection mechanism, merging multiple SQL insert statements for execution. If any statement fails, the entire process is rolled back. Furthermore, the application automatically performs database version checks and upgrades upon startup to ensure compatibility with older data.

10. A mobile multimodal information acquisition method according to claim 6, characterized in that, In step S4, the data interaction with external devices includes: the terminal reads local cached data and database fields, and automatically renders and generates a collection report in docx format according to the public security standard typesetting format. The report supports direct printing or export to external storage media. The terminal starts a background web server, and external devices access the terminal's IP address via the local area network to push the data collection progress in real time. The terminal also provides a packaged download interface to encapsulate the collected data of a specified case into a ZIP format for external devices to download and import into the backend platform.

Images