A driver physical examination intelligent auxiliary detection method and system based on a multi-modal AI

Abstract

The application discloses a kind of based on multimodal AI's driver physical examination intelligent auxiliary detection method and system, it is related to intelligent medical detection technical field, including following steps: S1, obtain identity data, identity portrait and physical examination data;S2, establish collection channel and detection item mapping and complete pre-processing;S3, obtain identity verification result by DeepFace model;S4, obtain physical examination item feature set by improving MediaPipe Pose model;S5, compare detection response and standard response information, obtain response comparison result;S6, fusion each detection result and compare with physical examination determination threshold, obtain item auxiliary detection result and item confidence;S7, according to operation mode forms auxiliary detection result or executes artificial review.The application adopts two-stage feature calibration layer and skeleton state backflow mechanism, realizes driver physical examination intelligent auxiliary detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical testing technology, and in particular to an intelligent auxiliary testing method and system for driver physical examination based on multimodal AI. Background Technology

[0002] As driver physical examination services move towards self-service, digitalization, and remote monitoring, intelligent auxiliary detection technologies that utilize general-purpose cameras, ID card readers, and audio equipment to complete identity verification and multi-item physical examinations have received widespread attention. Existing driver physical examination equipment mostly employs manual, item-by-item operation, a single visual model, or independent detection terminals, but in practical applications, the following problems are commonly encountered: When multiple cameras of the same model are connected, the device node number is prone to change depending on the startup order or hot-swapping. There is a lack of stable mapping between the acquisition channel and detection items such as height, posture, vision, and color recognition, which can easily lead to misalignment of image sources and misbinding of detection items. The image acquisition program and AI model usually run in the same process or transfer data through repeated file reading and writing, which can easily lead to image backlog and processing delays, making it difficult to meet the requirements of real-time acquisition of multiple images and parallel analysis of multiple models at the same time. Identity verification, posture detection, hand action recognition, vision and color recognition response, and hearing response are mostly judged independently. Differences in lighting and posture between ID photos and on-site faces, partial occlusion of key human body points, and single-frame drift can easily cause misjudgments. In addition, the existing system lacks item confidence, consistency verification by manual review, and discrepancy detection records, resulting in insufficient reliability and traceability of detection results.

[0003] Therefore, how to provide a driver physical examination intelligent auxiliary detection method and system based on multimodal AI is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose an intelligent auxiliary detection method and system for driver physical examination based on multimodal AI. This invention employs a fixed acquisition channel mapping, shared memory frame transmission, and multi-model AI fusion method to jointly detect identity information, human posture, visual color perception images, and auditory response. A two-stage feature calibration layer is introduced into the DeepFace model to perform overall calibration and local correction of identity facial features and on-site facial features. A hierarchical posture recursion layer and skeleton state backflow mechanism are introduced into the improved MediaPipePose model to recursively deduce and correct the state of key human body points. By fusing physical examination item features, response comparison results, and judgment thresholds, auxiliary detection results and item confidence levels are obtained. Results output and consistency verification are completed through a pure self-service mode and an AI-assisted manual mode. This method has the advantages of high detection efficiency, stable recognition, verifiable results, and traceable process.

[0005] According to an embodiment of the present invention, a driver physical examination intelligent auxiliary detection method based on multimodal AI includes the following steps: S1. Obtain the examinee's identity data, identity portrait, ID card surface image, and multimodal physical examination data; S2. Establish a mapping between the acquisition channel and the detection items, preprocess the multimodal physical examination data and transmit it to the AI ​​detection layer; S3. Input the identity image and the on-site face into the DeepFace model. The DeepFace model performs hierarchical extraction and difference calibration of facial features through a two-stage feature calibration layer, and obtains the identity verification result by combining text verification and liveness analysis. S4. Input the human body image into the improved MediaPipe Pose model. The improved MediaPipe Pose model recursively calculates the key point states of different human body regions through a layered pose recursion layer, and corrects the cross-layer key point states through a skeleton state backflow mechanism. Combined with the feature analysis of other physical examination images, a set of physical examination item features is obtained. S5. Obtain the test responses of the examinee in each test item, compare the test responses with the standard response information, and obtain the response comparison results; S6. Integrate the identity verification result, the physical examination item feature set, and the response comparison result, and compare them with the physical examination judgment threshold to obtain the item auxiliary detection result and the item confidence level; S7. Generate auxiliary detection results for driver physical examination based on the operating mode, or send the auxiliary detection results and confidence levels of the items to the doctor's terminal to obtain manual review results and perform consistency verification.

[0006] Optionally, S2 specifically includes: Traverse each image acquisition device node, read the device serial number and image acquisition capability identifier, filter out non-acquisition nodes according to the image acquisition capability identifier, and retain acquisition nodes with image acquisition capabilities; Each acquisition node is bound to the corresponding detection item according to the device serial number, forming a fixed mapping between the acquisition channel and the detection item; The C layer calls the acquisition node to obtain the original image frame, and performs pixel format conversion, size adjustment and image standardization processing required for the corresponding detection items on the original image frame to obtain the standard physical examination image frame. A shared memory region is established for each acquisition channel. The standard physical examination image frame is written into the corresponding shared memory region, along with the frame number and acquisition time. The AI ​​detection layer reads the latest standard physical examination image frame from each shared memory region according to the frame sequence number, discards the backlog of image frames with frame sequence numbers earlier than the latest standard physical examination image frame, and obtains the multimodal physical examination data to be detected.

[0007] Optionally, S3 specifically includes: Read the identity field from the identity data, perform text recognition on the ID card surface image to obtain the on-site identity field, and compare the identity field with the on-site identity field item by item to obtain the text verification result; Extract the on-site face region from continuous frontal human images, and input the identity portrait and the on-site face region into the DeepFace model to obtain the identity face features and the on-site face features respectively; The DeepFace model processes the identity face features and the on-site face features through a two-stage feature calibration layer. First, it performs overall structural calibration on the two types of face features to obtain the first-level calibration features. Then, it performs residual correction on the local differences in the first-level calibration features to obtain the second-level calibration features. Calculate the feature distance between the two types of secondary calibration features, and obtain the face comparison score based on the feature distance; Changes in eye state, head posture, and facial texture are extracted from continuous frontal human images. Liveness analysis results are obtained by comparing each change with the corresponding liveness threshold. The identity verification result is obtained based on the text verification result, the face comparison score, and the liveness analysis result.

[0008] Optionally, the improved MediaPipe Pose model includes an image feature encoding module, a hierarchical pose recursion layer, a skeleton state reflow mechanism, and a key point decoding module; The image feature encoding module performs staged convolution processing on continuous human images to obtain pose feature maps corresponding to each image frame. The layered posture recursion layer divides the posture feature map into head and neck posture features, upper limb posture features, and trunk and lower limb posture features according to the connection relationship of the human skeleton. Within each type of posture feature, the key point state is recursively passed according to the key point connection order, and the various posture features are spliced ​​together to obtain the skeleton recursion feature. The skeleton state backflow mechanism extracts the deep key point states from the skeleton recursive features, returns the deep key point states to the corresponding previous pose processing stage, and performs difference correction with the previous key point states to obtain backflow correction features. The key point decoding module performs coordinate mapping on the reflow correction features to obtain the coordinates of human posture key points in each image frame, and arranges the human posture key point coordinates according to the acquisition order of the image frames to obtain continuous posture states.

[0009] Optionally, the skeleton state reflow mechanism specifically includes: The deep key point states of each human body key point are extracted from the recursive features of the skeleton, and the correspondence between the deep key point states and the previous key point states is established according to the human body key point identifier. The state of each deep key point is passed to the corresponding preceding attitude processing stage along the return path opposite to the attitude recursion direction. Calculate the state difference between the state of the deep key point and the state of the corresponding preceding key point, and sort the key points of each human body according to the absolute value of the state difference to obtain the key point offset order. Human key points whose state difference exceeds the reflux threshold are identified as key points to be corrected. According to the key point offset order, the state of the corresponding deep key point is superimposed with the state of the previous key point to obtain the reflux state of the key point to be corrected. Based on the states of adjacent key points connected to the key point to be corrected, the return state is constrained by a skeleton to obtain the state of the corrected key point. The corrected key point state replaces the corresponding preceding key point state and is combined with the human body key point state that has not reached the backflow threshold to obtain the backflow correction feature.

[0010] Optionally, S5 specifically includes: Obtain the standard response information corresponding to each test item, and establish a response time window according to the execution order of the test items; The vision chart image is subjected to grayscale conversion and contour extraction, and the actual opening direction of the vision chart characters is determined based on the pixel distribution within the character area; The MediaPipe Hands model was used to extract the coordinates of key points of the hand from the frontal human image of the subject, and the direction of the subject's indication was determined based on the direction vector formed from the wrist key point to the fingertip key point. The visual response comparison result is obtained by comparing the direction indicated by the examinee with the actual opening direction of the characters on the visual chart. The test responses submitted by the test subjects within the corresponding response time windows of the colorimetry test and hearing test are obtained. Each test response is compared with the standard answer of the colorimetry chart and the pure tone playback channel to obtain the colorimetry response comparison results and the hearing response comparison results. The visual response comparison results, color recognition response comparison results, and hearing response comparison results are summarized according to the test item labels to obtain the response comparison results.

[0011] Optionally, the step of obtaining the test responses submitted by the subject within the corresponding response time windows of the colorimetry and hearing tests, and comparing each test response with the standard answer of the colorimetry chart and the pure tone playback channel to obtain the colorimetry response comparison results and the hearing response comparison results, specifically: Extract colorimetric detection maps from the standard colorimetric map library and read the standard answers corresponding to the colorimetric detection maps; Extract the image features of the colorimetric spectrum image, compare the image features with the image features of each spectrum in the standard colorimetric spectrum library, and determine the actual colorimetric detection spectrum to be displayed based on the similarity ranking result; The test subject submits color discrimination answers via gesture or input within the corresponding response time window of the color discrimination ability test. The color discrimination answers are compared with the standard answers. The color discrimination response comparison result is obtained based on the number of correct responses within a preset number of questions. A sine wave audio signal containing fade-in and fade-out envelopes is generated through a pure tone generation program. The sine wave audio signal is played to the left and right channels respectively according to a preset frequency, and the pure tone playback channel, pure tone frequency and playback time are recorded. The hearing response submitted by the subject through gesture or input operation within the corresponding response time window of the hearing test is obtained, and the channel direction indicated by the hearing response is compared with the pure tone playback channel. The number of correct responses for the left and right channels at multiple frequencies was counted to obtain the hearing response comparison results.

[0012] Optionally, S6 specifically includes: According to the test item identifier, the identity verification result, the physical examination item feature set, and the response comparison result are respectively associated with the corresponding test item; The measurement values, action status, and duration of each test item are extracted from the feature set of the physical examination items, and invalid test data are filtered out based on the number of valid acquisition frames for each test item. Read the physical examination judgment threshold corresponding to each test item, and compare the measured value, the action state, the duration and the response comparison result with the corresponding physical examination judgment threshold to obtain the item judgment value of each test item; The face comparison score is compared with the identity verification threshold, and combined with the text verification result and the liveness analysis result to obtain the identity item judgment value; The model detection confidence value and response accuracy of each detection item are normalized according to the effective acquisition frame ratio, model detection confidence value and response accuracy, and the average value of the normalization results is calculated to obtain the item confidence of each detection item. By summarizing the item judgment value, the identity item judgment value, and the item confidence level according to the detection item identifier, the auxiliary detection result and the item confidence level are obtained.

[0013] Optionally, S7 specifically includes: Read the system operating mode and determine the pure self-service testing process or the AI-assisted manual testing process based on the system operating mode; In the fully self-service testing process, the auxiliary testing results and confidence levels of the tested items are summarized according to the testing item identifiers to form the driver's physical examination auxiliary testing results; In the AI-assisted manual detection process, the auxiliary detection results of the item, the confidence level of the item, and the corresponding detection evidence are sent to the doctor's terminal; The manual review results and electronic confirmation information submitted by the doctor's terminal are obtained, and the manual review results are compared with the corresponding auxiliary test results item by item to obtain the consistency verification results. When the manual review result is consistent with the auxiliary test result of the project, the manual review result is used to form the auxiliary test result of the driver's physical examination; When the manual review result is inconsistent with the auxiliary test result of the item, the discrepancy test item, the auxiliary test result of the item, the manual review result and the reason for the discrepancy are recorded, and a special case mark is formed; The driver's physical examination auxiliary test results, the consistency verification results, and the test process data are associated with the examinee's identity identifier to form a driver's physical examination record.

[0014] According to an embodiment of the present invention, a driver physical examination intelligent auxiliary detection system based on multimodal AI includes the following modules: The image acquisition module acquires the examinee's identity data, identity portrait, and ID card surface image through the identity reading device, and collects multimodal physical examination data through at least four universal RGB cameras of the same model; The preprocessing module establishes a fixed mapping between each general RGB camera and the detection item based on the camera device serial number and image acquisition capability identifier. It acquires each image through the C layer, performs format conversion and size adjustment on each image, and passes the latest image frame to the AI ​​detection layer through shared memory. The multi-model AI detection engine inputs the identity portrait and the on-site face into the DeepFace model, performs hierarchical extraction and difference calibration of facial features through a two-stage feature calibration layer, and obtains the identity verification result by combining text verification and liveness analysis; it inputs the human body image into the improved MediaPipe Pose model, recursively calculates the state of human body key points through a layered pose recursion layer, and corrects the state of cross-layer key points through a skeleton state backflow mechanism, and obtains the feature set of physical examination items by combining hand movements, head movements, vision and color discrimination analysis, and obtains the test subject's detection response in the corresponding test items; The hearing test module outputs a pure tone signal of a preset frequency through an audio device, obtains the test response submitted by the subject within the response time window, and compares the test response with the pure tone playback channel to obtain the hearing response comparison result. The fusion judgment engine integrates the identity verification results, the feature set of the physical examination items, and the response comparison results corresponding to each test item, and compares them with the physical examination judgment threshold to obtain the auxiliary test results and the confidence level of the items. The dual-track output module generates driver physical examination auxiliary test results based on the auxiliary test results of the project in the pure self-service mode, and sends the auxiliary test results of the project, the confidence level of the project and the test evidence to the doctor's terminal in the AI-assisted human mode to obtain the manual review results and perform consistency verification. The cloud-based monitoring platform performs encrypted transmission, resume interrupted transmission, remote configuration management, and full-process log recording on the driver's physical examination auxiliary detection results and detection process data.

[0015] The beneficial effects of this invention are: (1) The present invention filters acquisition nodes according to the camera device serial number and image acquisition capability identifier, establishes a fixed mapping between four general RGB cameras of the same model and the detection items, and completes image acquisition, format conversion and size adjustment through the C layer. It uses shared memory to transmit the latest image frame to the AI ​​detection layer according to the frame sequence number, reduces channel misalignment and image backlog caused by changes in device nodes, and improves the continuity and real-time performance of multi-channel physical examination image acquisition. (2) The overall structure calibration and local residual correction of identity face features and on-site face features are performed through a two-stage feature calibration layer. The identity verification results are formed by combining the ID card text verification, eye state changes, head posture changes and facial texture changes. This reduces the impact of differences in brightness, angle and local occlusion between the ID card photo and the on-site image on the face comparison score and improves the accuracy of identity verification. (3) By recursively proposing key point states according to the skeletal connection relationship of the head and neck, upper limbs and trunk and lower limbs through the layered posture recursion layer, and correcting the offset key points according to the state difference and skeletal connection constraints through the skeletal state return mechanism, the stability of height, neck movement and upper limb movement detection is improved; further, the characteristics of physical examination items, visual color discrimination and hearing response and physical examination judgment threshold are integrated to obtain the auxiliary detection results and item confidence, and the manual review results, reasons for discrepancies and detection evidence are recorded to realize the complete record and result traceability of the driver's physical examination process. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a driver physical examination intelligent auxiliary detection method based on multimodal AI proposed in this invention; Figure 2This is a structural diagram of the improved MediaPipe Pose model proposed in this invention; Figure 3 This is a schematic diagram of the structure of a driver physical examination intelligent auxiliary detection system based on multimodal AI proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-2 A driver physical examination intelligent auxiliary detection method based on multimodal AI includes the following steps: S1. Obtain the examinee's identity data, identity portrait, ID card surface image, and multimodal physical examination data; S2. Establish a mapping between the data acquisition channel and the detection items, preprocess the multimodal physical examination data and transmit it to the AI ​​detection layer; S3. Input the identity image and the on-site face into the DeepFace model. The DeepFace model performs hierarchical extraction and difference calibration of facial features through a two-stage feature calibration layer, and obtains the identity verification result by combining text verification and liveness analysis. S4. Input the human body image into the improved MediaPipe Pose model. The improved MediaPipe Pose model recursively calculates the key point states of different human body regions through a layered pose recursion layer, and corrects the cross-layer key point states through a skeleton state backflow mechanism. Combined with the feature analysis of other physical examination images, the feature set of physical examination items is obtained. S5. Obtain the test responses of the examinee in each test item, compare the test responses with the standard response information, and obtain the response comparison results; S6. Integrate the identity verification results, the feature set of physical examination items, and the response comparison results, and compare them with the physical examination judgment threshold to obtain the auxiliary detection results and the confidence level of the items; S7. Generate auxiliary test results for driver physical examination based on the operating mode, or send the auxiliary test results and confidence level of the project to the doctor's terminal to obtain the manual review results and perform consistency verification.

[0019] In this embodiment, S2 specifically refers to: Iterate through each image acquisition device node, read the device serial number and image acquisition capability identifier, filter out non-acquisition nodes based on the image acquisition capability identifier, and retain acquisition nodes with image acquisition capabilities; Each data acquisition node is bound to the corresponding detection item according to the device serial number, forming a fixed mapping between the data acquisition channel and the detection item; The C layer calls the acquisition node to obtain the original image frame, performs pixel format conversion, size adjustment and image standardization processing required for the corresponding detection items on the original image frame to obtain the standard physical examination image frame; A shared memory region is established for each acquisition channel. Standard physical examination image frames are written into the corresponding shared memory region, along with the frame number and acquisition time. The AI ​​detection layer reads the latest standard physical examination image frame from each shared memory area according to the frame sequence number, discards the backlog image frames whose frame sequence number is earlier than the latest standard physical examination image frame, and obtains the multimodal physical examination data to be detected. In practice, the driver's physical examination terminal is equipped with four EMEET SmartCam C960 4K universal RGB cameras. The front acquisition channel is installed at the height marking position to capture images of the subject's frontal face, full-body posture, and hand movements. The side acquisition channel is positioned at a 45° angle to the subject's frontal direction to capture images of the head, neck, and side posture. The vision chart acquisition channel is fixedly aligned with a paper standard logarithmic vision chart. The color recognition acquisition channel is fixedly aligned with the colorimetric display area. The acquisition resolution of the four cameras is set to 1920×1080 pixels, the target frame rate is set to 30fps, and the actual operating frame rate is maintained between 25 and 30fps.

[0020] After system startup, the system iterates through the / dev / video* device nodes, reads the ID_SERIAL field of each node using udevadm info, and checks whether ID_V4L_CAPABILITIES contains :capture:. Metadata nodes that do not contain the capture capability identifier do not establish capture mappings. For the retained capture nodes, symbolic links / dev / camera_front, / dev / camera_side, / dev / camera_chart, and / dev / camera_color are created according to the device serial number. At the same time, the device serial number and symbolic link name are written to / etc / udev / rules.d / 99-cameras.rules, thus forming a fixed mapping between capture channels and detection items, avoiding capture channel misalignment caused by changes in node numbers after hot-plugging devices.

[0021] The C layer, based on the V4L2 framework, configures image size, pixel format, and frame rate via ioctl and reads raw image frames using mmap. It converts the YUYV format to BGR format according to the rule that every two pixels share a set of chromaticity components, and then adjusts it to 1280×720 pixels using bilinear interpolation. Pixel value normalization is performed on frontal and side images; grayscale conversion and contrast stretching are performed on the visual acuity chart images; and white balance correction and standard color temperature compensation are performed on the colorimetric images. The processed images form standard physical examination image frames.

[0022] Four acquisition channels establish shared memory regions: / dev / shm / das_frame_front, / dev / shm / das_frame_side, / dev / shm / das_frame_chart, and / dev / shm / das_frame_color. For each frame written, the frame number is incremented by 1 from the previous frame, and the acquisition time (in milliseconds) is recorded synchronously. The AI ​​detection layer compares the frame number in the shared memory with the previously read frame number, reading image data only when the frame number increases. When multiple unprocessed frames exist simultaneously, the image frame with the largest frame number is retained. The main business environment runs MediaPipe models, while the AI ​​vision environment runs RetinaFace and DeepFace models. Both environments read the same batch of latest image frames through shared memory, forming multimodal body examination data to be detected. This ensures that the end-to-end latency from acquisition to AI detection result return for the four images is controlled within 200ms.

[0023] In this embodiment, S3 specifically refers to: Read the identity field from the identity data, perform text recognition on the surface image of the ID card to obtain the on-site identity field, and compare the identity field with the on-site identity field item by item to obtain the text verification result; Extract the on-site face region from continuous frontal human images, input the identity portrait and the on-site face region into the DeepFace model, and obtain the identity face features and the on-site face features respectively; The DeepFace model processes identity-based facial features and on-site facial features through a two-stage feature calibration layer. First, it performs overall structural calibration on the two types of facial features to obtain first-level calibration features. Then, it performs residual correction on the local differences in the first-level calibration features to obtain second-level calibration features. Calculate the feature distance between the two types of secondary calibration features, and obtain the face comparison score based on the feature distance; Changes in eye state, head posture, and facial texture are extracted from continuous frontal human images. Liveness analysis results are obtained by comparing each change with the corresponding liveness threshold. The identity verification result is obtained based on the text verification result, the face comparison score, and the liveness analysis result; In the specific implementation process, the identity reading device uses a USB second-generation ID card reader. After reading the ID card chip, it generates an identity data file containing fields for name, ID card number, date of birth, and address, as well as an identity image with a resolution of 358×441 pixels. Simultaneously, an image acquisition device with a resolution of no less than 1200×800 pixels is used to acquire an image of the ID card surface, and pytesseract is used to perform grayscale conversion, binarization, character region segmentation, and text recognition. After removing spaces and non-text characters from each on-site identity field, it is compared item by item with the corresponding identity field. If the name and ID card number match and the consistency rate of the remaining fields is no less than 80%, the text verification result is recorded as passed.

[0024] At least 30 frames of human images are continuously acquired from the front acquisition channel. RetinaFace is used to extract the largest face region with a confidence level of at least 0.90. The identity portrait and on-site face regions are adjusted to 224×224 pixels and input into the DeepFace model to extract 512-dimensional identity face features and 512-dimensional on-site face features. A two-stage feature calibration layer first calculates the mean and standard deviation of the two types of face features respectively. Then, each feature dimension is subtracted from its corresponding mean and divided by the standard deviation and 10⁻⁻⁶. 6 The sum of the values ​​is then subjected to L2 normalization to obtain the first-level calibration features. Subsequently, the first-level calibration features are divided into 8 groups, each containing 64-dimensional features. The absolute difference between the two types of features in each group is calculated. The feature group with an average difference greater than 0.12 within the group is determined as the local difference group. The difference between the identity first-level calibration features and the field first-level calibration features is superimposed on the corresponding feature group according to a residual coefficient of 0.3 to obtain the second-level calibration features.

[0025] The cosine distance between the two types of secondary calibration features is calculated, and the feature similarity is obtained by subtracting the cosine distance from one. The feature similarity is then multiplied by 100 to obtain the face comparison score. A face comparison score of 60 or higher is considered a passed face comparison. Liveness analysis uses a 3-second detection window. A blink is counted when the eye aspect ratio first falls below 0.20 and then recovers to above 0.25. A head yaw angle exceeding 10° is counted as a valid pose change. A high-frequency texture energy change rate exceeding 8% in consecutive frames is counted as a valid texture change. At least two of the three liveness conditions are met for the liveness analysis result to be considered passed. An identity verification pass result is obtained when the text verification result, face comparison result, and liveness analysis result are all passed; otherwise, a result requiring manual review is generated. Through two-stage feature calibration, the impact of differences in lighting, shooting angle, and local occlusion between the ID photo and the on-site image on the face comparison score can be reduced.

[0026] In this embodiment, the improved MediaPipe Pose model includes an image feature encoding module, a hierarchical pose recursion layer, a skeleton state reflow mechanism, and a key point decoding module. The image feature encoding module performs staged convolution processing on continuous human images to obtain pose feature maps corresponding to each image frame; The hierarchical posture recursive layer divides the posture feature map into head and neck posture features, upper limb posture features, and trunk and lower limb posture features according to the connection relationship of the human skeleton. Within each type of posture feature, the key point state is recursively passed according to the key point connection order, and the various posture features are spliced ​​together to obtain the skeleton recursive features. The skeleton state reflow mechanism extracts the deep key point states from the skeleton recursive features, returns the deep key point states to the corresponding previous pose processing stage, and performs difference correction with the previous key point states to obtain the reflow correction features. The key point decoding module performs coordinate mapping on the reflow correction features to obtain the coordinates of human pose key points in each image frame, and arranges the human pose key point coordinates according to the acquisition order of the image frames to obtain continuous pose states. In the specific implementation process, the improved MediaPipe Pose model retains the processing framework of the traditional MediaPipe Pose model for keypoint detection in continuous human images, and adds a hierarchical pose recursion layer and a skeleton state reflow mechanism between the image feature encoding module and the keypoint decoding module. First, the continuous human images are adjusted to 256×256 pixels, and mean removal and standard deviation normalization are performed on the RGB three channels to obtain standardized human images. The image feature encoding module uses a convolutional layer with a stride of 2 and residual convolutional units to perform four-stage feature extraction, with the number of output channels set to 32, 64, 128, and 256 for each stage, resulting in pose feature maps with progressively decreasing resolution. A 1×1 convolution is used to uniformly map the number of channels in each pose feature map to 128, and the corresponding 128-dimensional keypoint states are extracted based on the predicted positions of human keypoints.

[0027] The hierarchical pose recursive layer divides keypoint states into head and neck pose features, upper limb pose features, and trunk and lower limb pose features according to the connection relationships of the human skeleton. The three types of pose features are recursively deduced in the order of connection from nose to shoulder, shoulder to wrist, and shoulder midpoint to ankle, respectively. During each recursion, the current keypoint state is added dimension-wise to the previous connected keypoint state and multiplied by 0.5, and the updated keypoint state is obtained through linear mapping. After completing the recursion within the region, the three types of pose features are concatenated in sequence to obtain the skeleton recursive features.

[0028] The skeleton state reflow mechanism extracts deep keypoint states from the recursive skeleton features and returns them to the corresponding preceding pose processing stage for difference correction, resulting in reflow-corrected features. The keypoint decoding module inputs the reflow-corrected features into the coordinate regression layer to obtain the x-coordinate, y-coordinate, and detection confidence value of each keypoint. The normalized coordinates are then multiplied by the image width and height to convert them into pixel coordinates. Keypoints with a detection confidence value below 0.6 within 30 consecutive frames are considered invalid keypoints, and the remaining keypoints are arranged according to the acquisition time to obtain continuous pose states. Compared to the unmodified MediaPipe Pose model, which mainly relies on frame-by-frame local feature decoding, this model uses regional recursion and deep state reflow to ensure continuous transmission of skeleton constraints across different human body regions. Even with partial occlusion or single-frame keypoint offset, it maintains pose state coherence, thereby improving the stability of height measurement, upper limb motion detection, and posture judgment.

[0029] In this embodiment, the skeleton state reflow mechanism is specifically as follows: The deep key point states of each human body key point are extracted from the recursive features of the skeleton, and the correspondence between the deep key point states and the previous key point states is established according to the human body key point identifier. The state of each deep key point is passed to the corresponding preceding attitude processing stage along the return path opposite to the attitude recursion direction. Calculate the state difference between the state of the deep keypoint and the state of the corresponding preceding keypoint, and sort the keypoints of each human body according to the absolute value of the state difference to obtain the keypoint offset order. Human key points whose state difference exceeds the reflux threshold are identified as key points to be corrected. According to the key point offset order, the state of the corresponding deep key point is superimposed with the state of the previous key point to obtain the reflux state of the key point to be corrected. Based on the states of adjacent key points connected to the key point to be corrected, skeleton connection constraints are applied to the reflow state to obtain the state of the corrected key point. The corresponding preceding keypoint state is replaced by the corrected keypoint state, and combined with the human keypoint state that has not reached the reflux threshold to obtain the reflux correction feature. In its implementation, the skeleton state reflow mechanism receives the skeleton recursive features output from the hierarchical posture recursive layer and splits them into multiple 128-dimensional deep keypoint states according to the human keypoint identifiers. Simultaneously, it reads the 128-dimensional preceding keypoint states corresponding to the same keypoint identifier from the preceding posture processing stage, establishing a one-to-one correspondence based on the human keypoint identifiers. For each human keypoint, the deep keypoint state is transferred to the preceding posture processing stage along a reflow path opposite to the posture recursion direction. In the head and neck region, the reflow occurs from the shoulder to the ear, eye, and nose; in the upper limb region, it occurs from the wrist to the elbow and shoulder; and in the trunk and lower limb region, it occurs from the ankle to the knee, hip, and the midpoint of the shoulder.

[0030] For the same human keypoint, the deep keypoint state is subtracted dimension-by-dimensional from the preceding keypoint state to obtain a 128-dimensional state difference. The absolute values ​​of the state differences in each dimension are then averaged to obtain the state offset value of the human keypoint. All human keypoints are then arranged in descending order of state offset value to form a keypoint offset order. The reflow threshold is set to 0.12. Human keypoints with a state offset value greater than 0.12 are identified as keypoints to be corrected, while the states of the preceding keypoints are retained for the remaining human keypoints. For keypoints to be corrected, they are processed sequentially according to the keypoint offset order. The state difference is multiplied by 0.4 to obtain the residual correction amount, which is then added dimension-by-dimensionally to the preceding keypoint state to obtain the reflow state.

[0031] After residual overlay is completed, the states of adjacent keypoints directly connected to the keypoint to be corrected are read from the human skeleton connection table. The normalized distance between the keypoint to be corrected and its adjacent keypoints is calculated and compared with the average length of the corresponding bone segment over 30 consecutive frames. When the distance deviation ratio exceeds 20%, the reflow state is multiplied by 0.7, the mean of the adjacent keypoint states is multiplied by 0.3, and then the two are added dimension by dimension to obtain the corrected keypoint state. When the distance deviation ratio does not exceed 20%, the reflow state is directly used as the corrected keypoint state. Finally, the corrected keypoint state replaces the corresponding preceding keypoint state, and the states of human keypoints that have not reached the reflow threshold are rearranged according to the human keypoint identifier to obtain the reflow correction feature. Compared to the traditional MediaPipe Pose model where deep features are directly passed to the keypoint decoding module, the skeleton state backflow mechanism can use deep keypoint states to reverse correct the offset of preceding keypoints and suppress abnormal corrections that do not conform to human body connection relationships through bone segment length constraints. This makes the coordinate recovery of occluded or fast-moving keypoints more stable and reduces the cumulative impact of single-frame false detections on continuous pose states.

[0032] In this embodiment, S5 specifically refers to: Obtain the standard response information corresponding to each test item, and establish a response time window according to the execution order of the test items; The image of the vision chart is converted to grayscale and its contour is extracted. The actual opening direction of the characters in the vision chart is determined based on the pixel distribution within the character area. The MediaPipe Hands model was used to extract the coordinates of key points of the hand from the frontal human image of the subject, and the direction of the subject's indication was determined based on the direction vector formed from the wrist key point to the fingertip key point. The visual response comparison results are obtained by comparing the direction indicated by the examinee with the actual opening direction of the characters on the visual chart. The test responses submitted by the test subjects within the corresponding response time windows of the colorimetry test and hearing test are obtained. Each test response is compared with the standard answer of the colorimetry chart and the pure tone playback channel to obtain the colorimetry response comparison results and the hearing response comparison results. The visual response comparison results, color vision response comparison results, and hearing response comparison results are summarized according to the test item labels to obtain the response comparison results; In the specific implementation process, the system establishes response time windows according to the execution order of visual acuity test, color vision test, and hearing test. The response time window for a single visual acuity chart character is set to 5 seconds, the response time window for a color vision test chart is set to 15 seconds, and the response time window for a single pure tone signal is set to 3 seconds. Each response time window starts from the time when the corresponding visual acuity chart character, color vision test chart, or pure tone signal begins to be displayed, and records the time when the examinee submits the test response through gesture operation or input operation.

[0033] During vision testing, the CAMERA_CHART acquisition channel is fixedly aligned with the standard logarithmic visual acuity chart on paper, and the acquisition resolution is set to 1920×1080 pixels. After converting the visual acuity chart image to grayscale, an adaptive thresholding method is used for binarization, and then the outer contour of the character region is extracted. Interference regions with an area less than 30% of the average area of ​​the target character are filtered out based on the contour area, and the remaining character regions are divided into four statistical regions: upper, lower, left, and right. The number of white pixels in each of the four regions is counted, and the direction corresponding to the region with the fewest white pixels is determined as the actual opening direction of the visual acuity chart character.

[0034] CAMERA_FRONT acquires a frontal human image of the subject, and the MediaPipe Hands model extracts 21 key points of the hand. A direction vector is established by selecting the wrist key point and the index fingertip key point. The differences between the x-coordinate of the index fingertip and the wrist x-coordinate, and the differences between the index fingertip and the wrist y-coordinate are calculated. When the absolute value of the x-coordinate difference is greater than the absolute value of the y-coordinate difference, the direction of indication (left or right) is determined based on the sign of the x-coordinate difference; otherwise, the direction of indication (up or down) is determined based on the sign of the y-coordinate difference. The subject's indicated direction is compared sequentially with the actual opening direction of the characters on the vision chart. A correct response is recorded if the direction matches; an incorrect response is recorded if the direction does not match or if the response is not submitted within the response time window. Each vision level is tested a maximum of 5 times. A vision level is considered passed if there are at least 3 correct responses. The test proceeds sequentially from level 5.0 down, with each eye tested separately.

[0035] After the vision test is completed, the system records the vision response comparison results, color vision response comparison results, and hearing response comparison results according to the test item identifiers. Each result is associated with the corresponding test item identifier, number of responses, number of correct responses, and response time, and arranged according to the execution order of the test items to obtain the response comparison results.

[0036] In this embodiment, the test responses submitted by the subject within the corresponding response time windows of the colorimetry and hearing tests are obtained. Each test response is compared with the standard answer of the colorimetry chart and the pure tone playback channel to obtain the colorimetry response comparison results and the hearing response comparison results, specifically: Extract colorimetric detection maps from the standard colorimetric map library and read the standard answers corresponding to the colorimetric detection maps; Extract image features from the colorimetric spectrum image, compare the image features with the image features of each spectrum in the standard colorimetric spectrum library, and determine the actual colorimetric detection spectrum to be displayed based on the similarity ranking results; The tester obtains the color discrimination answers submitted by the subject through gesture or input operations within the corresponding response time window of the color discrimination ability test. The color discrimination answers are compared with the standard answers, and the color discrimination response comparison results are obtained based on the number of correct responses within the preset number of questions. A sine wave audio signal containing fade-in and fade-out envelopes is generated through a pure tone generation program. The sine wave audio signal is played to the left and right channels respectively according to a preset frequency, and the pure tone playback channel, pure tone frequency and playback time are recorded. Acquire the hearing response submitted by the subject through gesture or input operation within the corresponding response time window of the hearing test, and compare the channel direction indicated by the hearing response with the pure tone playback channel; The number of correct responses for the left and right channels at multiple frequencies was counted separately to obtain the hearing response comparison results. In the specific implementation process, the standard colorimetric spectrum library pre-stores no less than 30 standard colorimetric spectra, each of which is assigned a unique spectrum number and associated with the corresponding standard answer in the reference answer library; for each colorimetric ability test, 5 colorimetric test spectra are randomly selected from the standard colorimetric spectrum library. During random selection, the spectrum numbers that have already been selected are excluded to avoid displaying the same colorimetric test spectra repeatedly in the same test.

[0037] CAMERA_COLOR collects colorimetric images of the actual display area, performs resizing, white balance correction, and color temperature compensation on the images, and then extracts color histograms and contour distribution features. The color histograms and contour distribution features are normalized and then concatenated to form image features. The cosine similarity between these image features and the features of each image in the standard colorimetric library is calculated, and the images are sorted from highest to lowest similarity. The standard colorimetric image with the highest similarity (not lower than 0.85) is selected as the actual displayed colorimetric detection image, and the standard answer is retrieved based on the corresponding image number. Within a 15-second response time window, the subject submits their colorimetric answer via gestures or input. The colorimetric answer is compared item by item with the standard answer; a match is recorded as a correct response. If there are at least 3 correct responses out of the 5 colorimetric detection images, the colorimetric response comparison result is recorded as passing; otherwise, it is recorded as failing.

[0038] The hearing test uses the Sounddevice pure tone generation program to generate sinusoidal audio signals in the order of 500Hz, 1kHz, 2kHz, 3kHz, and 4kHz, with each audio signal lasting 2 seconds. A 100ms fade-in envelope and a 100ms fade-out envelope are set at the start and end of the audio signal, respectively, to suppress instantaneous peaks by gradually increasing or decreasing the signal amplitude. The system randomly selects either the left or right channel to play the pure tone signal and simultaneously records the pure tone playback channel, frequency, and playback time.

[0039] Within 3 seconds of the start of the pure tone playback, the subject submits a hearing response via gesture or input. The left or right direction indicated by the hearing response is compared with the pure tone playback channel. A correct response is recorded if the direction matches, and an incorrect response is recorded if the direction does not match or if a hearing response is not submitted within 3 seconds. The left and right channels are independently tested at multiple preset frequencies. Each channel is played 5 times. If there are at least 3 correct responses, the hearing response comparison result for that channel is recorded as passing. Finally, the number of correct responses, the number of incorrect responses, and the corresponding pure tone frequency are summarized according to the channel identifier to obtain the hearing response comparison result.

[0040] In this embodiment, S6 specifically refers to: According to the test item identifier, the identity verification result, the physical examination item feature set, and the response comparison result are respectively associated with the corresponding test item; Extract the measurement values, action status, and duration of each test item from the feature set of the physical examination items, and filter out invalid test data based on the number of valid acquisition frames for each test item; Read the physical examination judgment threshold corresponding to each test item, and compare the measurement value, action state, duration and response comparison results with the corresponding physical examination judgment threshold to obtain the item judgment value of each test item. The facial recognition score is compared with the identity verification threshold, and combined with the text verification results and liveness analysis results to obtain the identity item judgment value. The model detection confidence value and response accuracy of each detection item are normalized according to the effective acquisition frame ratio, model detection confidence value and response accuracy, and the average value of the normalization results is calculated to obtain the item confidence of each detection item. By summarizing the item judgment values, identity item judgment values, and item confidence scores according to the test item identifiers, we can obtain the auxiliary test results and item confidence scores. In the specific implementation process, the system sets separate test item labels for identity verification, height, vision, color vision, hearing, neck range of motion, and upper limb function. The identity verification results, physical examination item feature sets, and response comparison results are written into the same examinee's test record according to the test item labels. For image detection items, image frames in which key points or target areas are successfully identified within a single detection window are considered valid acquisition frames. The number of valid acquisition frames is divided by the total number of image frames acquired within that detection window to obtain the valid acquisition frame ratio. When the valid acquisition frame ratio is lower than 0.70, the corresponding detection data is filtered out and marked for re-acquisition.

[0041] Height detection extracts height measurements and the difference in longitudinal distance between the left and right ankles from the feature set of physical examination items. A standing posture is considered valid if the difference in longitudinal distance between the left and right ankles does not exceed 6.0 cm. Upper limb function detection extracts the normalized distance from fingertip to wrist. A distance less than 0.15 is considered a clenched fist, and a distance greater than 0.25 is considered an open hand. Both movements must be maintained continuously for at least 3 seconds for the item to pass. Neck mobility detection extracts the normalized x-coordinate of the key point at the tip of the nose. A x-coordinate less than 0.45 and maintained for at least 30 seconds is considered a successful left turn, and a x-coordinate greater than 0.55 and maintained for at least 30 seconds is considered a successful right turn. Visual acuity, color vision, and hearing tests count the number of correct responses. The response accuracy rate is calculated by dividing the number of correct responses by the total number of tests. A passing score is achieved if at least 3 out of 5 tests are correct.

[0042] In identity verification, if both text verification and liveness analysis pass and the face comparison score is not lower than 60 points, the identity item judgment value is recorded as passed. Item confidence is calculated based on the proportion of effective acquisition frames, the model detection confidence value, and the response accuracy, with each value limited to between 0 and 1. For items without response comparison data, the average of the proportion of effective acquisition frames and the model detection confidence value is used as the item confidence value. For items with response comparison data, the three values ​​are added together and divided by 3 to obtain the item confidence value. Finally, the judgment values ​​of each item, the identity item judgment value, and the corresponding item confidence values ​​are summarized according to the detection item identifier to form the item auxiliary detection results.

[0043] In this embodiment, S7 specifically refers to: Read the system operating mode and determine the pure self-service testing process or the AI-assisted manual testing process based on the system operating mode; In a fully self-service testing process, auxiliary test results and confidence levels of test items are summarized according to the test item identifiers to form auxiliary test results for driver physical examination. In the AI-assisted manual detection process, the auxiliary detection results, the confidence level of the project, and the corresponding detection evidence are sent to the doctor's terminal; Obtain the manual review results and electronic confirmation information submitted by the doctor's terminal, compare the manual review results with the corresponding auxiliary test results item by item, and obtain the consistency verification results; When the results of manual review are consistent with the results of auxiliary testing of the project, the results of manual review shall be used to form the auxiliary testing results of the driver's physical examination. When the results of manual review are inconsistent with the results of auxiliary tests, record the discrepancy test items, auxiliary test results, manual review results, and reasons for the discrepancy, and create a special case marker; The driver's physical examination auxiliary test results, consistency verification results and test process data are linked to the examinee's identity to form a driver's physical examination record; In the specific implementation process, when the system starts up, it reads the running mode field from the cloud configuration file. When the running mode field is set to self_service, it enters the pure self-service detection process; when it is set to ai_assisted, it enters the AI-assisted manual detection process. If the running mode field is missing or fails to be read, it defaults to the AI-assisted manual detection process and records the configuration reading error. In the pure self-service detection process, the system reads the auxiliary detection results and confidence levels of the detection items in sequence according to the detection item identifiers. It writes the judgment values, measurement values, and confidence levels of the corresponding items for identity verification, height, vision, color vision, hearing, neck range of motion, and upper limb function into a unified result file. When all mandatory detection items are completed and there is no need for re-collection and marking, the driver's physical examination auxiliary detection result is formed.

[0044] In the AI-assisted manual testing process, the system sends the auxiliary test results, confidence levels, and test evidence to the doctor's terminal. Test evidence includes identity verification scores, key image frames, key point coordinates, action duration, visual acuity and color perception response records, pure audio frequency, playback channels, and the examinee's response time. After selecting a test item, the doctor reviews the corresponding test evidence and submits a manual review result (pass, fail, or retest), simultaneously generating electronic confirmation information through electronic signatures or electronic seals.

[0045] The system compares the manual review results with the auxiliary test results item by item according to the test item identifier. When the two judgment values ​​are the same, the consistency check result of the corresponding item is recorded as consistent; when the judgment values ​​are different or the manual review result indicates retesting, the consistency check result is recorded as inconsistent. When all test items are consistent, the manual review result is used to form the driver's physical examination auxiliary test result; when there are inconsistent items, the system records the dissenting test item, the auxiliary test result, the manual review result, the reason for the discrepancy filled in by the doctor, and the corresponding test evidence, and sets the special case mark to 1. Finally, using the examinee's identity identifier as the association key, the driver's physical examination auxiliary test result, consistency check result, special case mark, timestamp of each test item, item confidence level, equipment status, and manual operation record are written into the driver's physical examination record.

[0046] refer to Figure 3 A driver physical examination intelligent auxiliary detection system based on multimodal AI includes the following modules: The image acquisition module acquires the examinee's identity data, identity portrait, and ID card surface image through the identity reading device, and collects multimodal physical examination data through at least four universal RGB cameras of the same model; The preprocessing module establishes a fixed mapping between each general RGB camera and the detection item based on the camera device serial number and image acquisition capability identifier. It acquires each image through the C layer, performs format conversion and size adjustment on each image, and passes the latest image frame to the AI ​​detection layer through shared memory. The multi-model AI detection engine inputs identity images and on-site faces into the DeepFace model. It performs hierarchical extraction and difference calibration of facial features through a two-stage feature calibration layer, and obtains identity verification results by combining text verification and liveness analysis. It inputs human images into the improved MediaPipe Pose model, and recursively calculates the state of human key points through a hierarchical pose recursion layer. It also corrects the state of key points across layers through a skeleton state backflow mechanism. It obtains the feature set of physical examination items by combining hand movements, head movements, vision and color recognition analysis, and obtains the test subject's detection response in the corresponding test items. The hearing test module outputs a pure tone signal of a preset frequency through an audio device, obtains the test response submitted by the test subject within the response time window, and compares the test response with the pure tone playback channel to obtain the hearing response comparison result. The fusion judgment engine integrates the identity verification results, the feature set of physical examination items, and the response comparison results corresponding to each test item, and compares them with the physical examination judgment threshold to obtain the auxiliary test results and the confidence level of the items. The dual-track output module generates driver physical examination auxiliary test results based on the project auxiliary test results in the pure self-service mode, and sends the project auxiliary test results, project confidence and test evidence to the doctor's terminal in the AI-assisted human mode to obtain the human review results and perform consistency verification. The cloud-based monitoring platform performs encrypted transmission, resume interrupted transmission, remote configuration management, and full-process log recording of driver physical examination auxiliary test results and test process data; In its implementation, the intelligent auxiliary detection system for driver physical examination is deployed in three layers: terminal layer, intelligent processing layer, and cloud monitoring layer. The terminal layer includes a USB second-generation ID card reader, four EMEET SmartCam C960 4K universal RGB cameras, dual-channel audio equipment, a patient interaction device, and a doctor's terminal. The front camera is installed at the height marking position to capture images of the frontal face, full-body posture, and hand movements. The side cameras are set at a 45° angle to the patient's front to capture images of head and neck movements. The other two cameras are aimed at the paper standard logarithmic visual acuity chart and the colorimetric chart display area, respectively. The four cameras have a resolution of 1920×1080 pixels and a frame rate of 30fps, with the actual operating frame rate maintained between 25 and 30fps.

[0047] The preprocessing module is deployed between the terminal layer and the intelligent processing layer. It filters acquisition nodes by device serial number and image acquisition capability identification, and establishes fixed device mappings using udev rules. The C layer, based on the V4L2 framework, acquires raw image frames through ioctl and mmap, converts YUYV format to BGR format, adjusts it to 1280×720 pixels, and then writes it to four independent shared memory areas. The multi-model AI detection engine runs in the intelligent processing layer. The main business environment runs MediaPipe Pose, MediaPipe Hands, and MediaPipe Face Mesh models, while the AI ​​vision environment runs RetinaFace and DeepFace models. Both environments read the latest image frame according to the frame sequence number and discard backlogged image frames, keeping the end-to-end latency from image acquisition to the return of project auxiliary detection results within 200ms.

[0048] The hearing test module generates pure tone signals of 500Hz, 1kHz, 2kHz, 3kHz, and 4kHz using the Sounddevice program. Each playback lasts for 2 seconds, with 100ms fade-in and fade-out envelopes set at the beginning and end. Within a 3-second response window, the subject submits the channel direction, which the system compares with the pure tone playback channel. The fusion judgment engine summarizes the identity verification results, physical examination item feature sets, and response comparison results according to the test item identifier, and calculates the item confidence level based on the effective acquisition frame ratio, model detection confidence value, and response accuracy.

[0049] The dual-track output module reads the operating mode field from the cloud configuration file. In pure self-service mode, it directly summarizes the results of each test item. In AI-assisted human mode, it sends the auxiliary test results, confidence levels, and key image frames to the doctor's terminal, and records the human review results, electronic confirmation information, and reasons for disagreement. The cloud monitoring platform uses HTTPS and JWT for transmission authentication, encrypts the driver's physical examination auxiliary test results with a 256-bit AES key, and then encrypts the AES key with a 2048-bit RSA public key. In case of network interruption, the data to be uploaded is cached locally, and the upload continues according to the test record number after the connection is restored. The cloud uses PostgreSQL to store driver's physical examination records, uses Redis to cache the running status and remote configuration parameters, and compresses and archives logs and test evidence according to a preset archiving period.

[0050] To verify the feasibility of this invention in practice, it was applied to a driver physical examination service terminal to test identity verification, joint detection of height and upper limb function, and AI-assisted manual dual-track detection process.

[0051] Example 1: Identity Verification Scenario In this embodiment, the examinee places their second-generation ID card in the sensing area of ​​a USB ID card reader. The reader uses a C program to read the name, ID number, address, issuing authority, and portrait image from the ID card chip, generating separate identity data and portrait image files. Simultaneously, an image acquisition device acquires an image of the ID card surface. The system performs grayscale conversion, binarization, and character region segmentation on the ID card surface image, and then uses Pytesseract to identify the name, ID number, and address fields. After removing spaces and non-text characters from the on-site identity fields, each field is compared with the chip identity fields. If the name and ID number are completely identical and the consistency rate of the remaining fields is not less than 80%, the text verification result is recorded as passed.

[0052] CAMERA_FRONT continuously acquired frontal human images of the subject. RetinaFace extracted the largest on-site face region with a detection confidence value of not less than 0.90 from the continuous images. The identity portrait and the on-site face region were adjusted to 224×224 pixels and input into the DeepFace model. The two-stage feature calibration layer first performed mean correction and L2 normalization on two types of 512-dimensional face features, then divided the features into 8 groups. Residual correction was performed on local features with an average difference within the group exceeding 0.12. The face comparison score was calculated according to the corrected cosine similarity. The face comparison score of a certain subject was 76.8 points, which is higher than the threshold of 60 points. Effective blinking, 12.4° head posture change and 10.6% facial texture change rate were detected in the continuous 3-second images. The liveness analysis result was recorded as passed, and the final identity verification result was recorded as passed.

[0053] The system of this invention was used to verify the identities of 120 subjects, and the identities were also verified manually by staff. The results are shown in Table 1.

[0054] Table 1. Detection Results of Identity Verification Scenarios

[0055] As shown in Table 1, the system of this invention achieves a 99.2% accuracy rate in recognizing identity fields, which is 1.7 percentage points higher than the manual entry and verification method. The initial identity verification pass rate increases from 93.3% to 98.3%. The average identity verification time of this system is 8.7 seconds, which is 22.5 seconds less than the 31.2 seconds required for manual verification, a reduction of approximately 72.1%. In the test, two subjects had initial face comparison scores of 58.7 and 59.4 respectively due to glare from their glasses and facial distortion on-site. The system did not directly provide a pass result but instead generated an identity verification result requiring manual review to avoid misjudgment of low-confidence data. After performing two-stage feature calibration on 18 sets of images with brightness differences, slight profile views, and local glare, the average face comparison score increased by 6.8 points, indicating that the two-stage feature calibration layer can reduce the differences between ID card photos and on-site faces in terms of lighting and shooting posture.

[0056] Example 2: Scenario of combined height and upper limb function testing In this embodiment, the subject stands in the detection area set in front of the CAMERA_FRONT. The horizontal distance between the camera and the subject is set to 2.2m, the camera installation height is set to 1.35m, and the acquisition resolution is set to 1920×1080 pixels. A fixed marker with an actual height of 100cm is set on one side of the detection area. The system reads the pixel height of the fixed marker in the image, divides 100cm by the corresponding pixel height to obtain the actual length corresponding to a single pixel, and then multiplies the pixel distance from the top of the head to the ankle by the actual length corresponding to a single pixel to obtain the height measurement value.

[0057] The improved MediaPipe Pose model extracts the coordinates of key points in human pose from continuous human images. Layered pose recursion layers process the key point states of the head and neck, upper limbs, and trunk and lower limbs respectively. A skeletal state reflow mechanism corrects for offset key points in the shoulders, hips, knees, and ankles. The subject is considered to be in a valid standing position when the longitudinal distance difference between the left and right ankles does not exceed 6.0 cm and the effective frame rate is not less than 0.70. A subject's system-measured height was 173.2 cm, while a standard height measurement device yielded 173.8 cm, a difference of 0.6 cm.

[0058] After height detection, the system uses the MediaPipe Hands model to extract 21 key hand points. The subject follows the terminal prompts to complete clenching and opening hand movements sequentially. In the clenched fist state, the normalized distance from fingertip to wrist is 0.12, less than the 0.15 threshold for clenching; in the open hand state, the normalized distance from fingertip to wrist is 0.31, greater than the 0.25 threshold for opening hand. Both movements are maintained continuously for more than 3 seconds, and the corresponding upper limb function assessment value is recorded as passing.

[0059] Eighty subjects were tested using both the unmodified model detection method and the system of this invention. Standard height measurement equipment and observation results of manual movements were used as the verification criteria, resulting in Table 2.

[0060] Table 2. Results of Combined Height and Upper Limb Function Tests

[0061] As shown in Table 2, the average absolute error of height in the system of this invention is 0.64 cm, which is 0.18 cm lower than that of conventional detection methods, and the maximum absolute error is controlled within 0.93 cm. The number of subjects who can effectively stand for the first time has increased from 72 to 76. The four subjects who did not complete the first collection mainly had problems with their feet being outside the image range or their standing position being off-center. After being prompted by the terminal to adjust their position, they all completed the detection. The first recognition rate of upper limb movements reached 96.3%, which is 5 percentage points higher than that of manual observation and recording. The three subjects who did not succeed in the first recognition were correctly recognized after re-performing the fist-clenching and hand-opening movements. The average time for joint detection was reduced from 51.8 seconds to 24.6 seconds, a reduction of about 52.5%. The above results show that the layered posture recursive layer can maintain the continuous relationship between key points in each human body region, and the skeletal state backflow mechanism can use the deep key point state to correct the offset of the previous key point, thereby reducing the impact of local occlusion and slight standing sway on height measurement and movement recognition.

[0062] Example 3: AI-assisted manual dual-track detection scenario In this embodiment, the system operates in an AI-assisted human mode. After identity verification, the examinee undergoes tests in the following order: height, visual acuity, color vision, hearing, neck mobility, and upper limb function. Visual acuity testing starts at level 5.0, with each eye tested separately. A maximum of five characters from the visual acuity chart are displayed for each visual acuity level, and passing the level requires at least three correct responses. Color vision testing involves randomly selecting five standard color charts from at least 30 charts, with a 15-second response time window for each chart. Passing requires at least three correct responses. Hearing testing involves playing pure tone signals at 500Hz, 1kHz, 2kHz, 3kHz, and 4kHz, each lasting two seconds. The examinee submits the vocal tract direction within a three-second response time window.

[0063] After the AI-assisted detection is completed, the doctor's terminal displays the auxiliary test results, confidence levels, and corresponding test evidence. One examinee's results were: left eye 4.9, right eye 5.0, color vision passed, hearing in both ears passed, neck range of motion passed, and upper limb function passed. The confidence levels for each test item ranged from 0.91 to 0.98. The doctor then accessed the hearing test review interface, checked the pure audio frequency, playback channels, playback time, and the examinee's response time, confirmed the test record, and submitted the manual review results and electronic confirmation information. The system compared the manual review results with the auxiliary test results item by item. Since all test items were consistent, a driver's physical examination record was generated.

[0064] The system of this invention was used to perform AI-assisted manual dual-track detection on 150 subjects, and compared with the traditional manual item-by-item detection process, resulting in Table 3.

[0065] Table 3. Implementation Results of Dual-Track Testing Process

[0066] As shown in Table 3, the average time for a single physical examination using the dual-track detection process of this invention is 3.9 minutes, a reduction of 6.8 minutes compared to the 10.7 minutes of the traditional manual detection process, representing a reduction in detection time of approximately 63.6%. In the traditional manual detection process, the initial detection conclusions of 138 examinees were consistent with the final review conclusions, with a consistency rate of 92.0%. In the dual-track detection process of this invention, the auxiliary test results of 145 examinees were consistent with the manual review results, achieving a consistency rate of 96.7%. Among the remaining 5 examinees, 3 had discrepancies in hearing response due to environmental noise, and 2 required re-testing due to unstable visual indication direction. The system recorded the discrepancies in the test items, the auxiliary test results, the manual review results, the reasons for the discrepancies, and the corresponding test evidence, and formed special case markers.

[0067] The four cameras operated at an average frame rate of 27.4fps during testing, with an average processing latency of 176ms from image acquisition to the return of auxiliary detection results. No sustained delays due to image backlog were observed. Traditional manual inspection processes primarily record the final conclusions, but some key image frames, response times, and device status are not saved, resulting in a data integrity record rate of only 68.7%. This invention links key image frames, key point coordinates, the number of correct responses, pure tone playback channels, item confidence levels, manual review results, and electronic confirmation information to the examinee's identity identifier, achieving a 100% data integrity record rate and 100% traceability rate for disputed items. Therefore, this invention shortens driver physical examination time while preserving complete evidence of AI detection and manual review, solving the problems of low efficiency, inconsistent judgment standards, and difficulty in tracing disputed results in traditional manual inspections.

[0068] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A driver physical examination intelligent auxiliary detection method based on multimodal AI, characterized in that, Includes the following steps: S1. Obtain the examinee's identity data, identity portrait, ID card surface image, and multimodal physical examination data; S2. Establish a mapping between the acquisition channel and the detection items, preprocess the multimodal physical examination data and transmit it to the AI ​​detection layer; S3. Input the identity image and the on-site face into the DeepFace model. The DeepFace model performs hierarchical extraction and difference calibration of facial features through a two-stage feature calibration layer, and obtains the identity verification result by combining text verification and liveness analysis. S4. Input the human body image into the improved MediaPipe Pose model. The improved MediaPipe Pose model recursively calculates the key point states of different human body regions through a layered pose recursion layer, and corrects the cross-layer key point states through a skeleton state backflow mechanism. Combined with the feature analysis of other physical examination images, a set of physical examination item features is obtained. S5. Obtain the test responses of the examinee in each test item, compare the test responses with the standard response information, and obtain the response comparison results; S6. Integrate the identity verification result, the physical examination item feature set, and the response comparison result, and compare them with the physical examination judgment threshold to obtain the item auxiliary detection result and the item confidence level; S7. Generate auxiliary detection results for driver physical examination based on the operating mode, or send the auxiliary detection results and confidence levels of the items to the doctor's terminal to obtain manual review results and perform consistency verification.

2. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 1, characterized in that, Specifically, S2 is: Traverse each image acquisition device node, read the device serial number and image acquisition capability identifier, filter out non-acquisition nodes according to the image acquisition capability identifier, and retain acquisition nodes with image acquisition capabilities; Each acquisition node is bound to the corresponding detection item according to the device serial number, forming a fixed mapping between the acquisition channel and the detection item; The C layer calls the acquisition node to obtain the original image frame, and performs pixel format conversion, size adjustment and image standardization processing required for the corresponding detection items on the original image frame to obtain the standard physical examination image frame. A shared memory region is established for each acquisition channel. The standard physical examination image frame is written into the corresponding shared memory region, along with the frame number and acquisition time. The AI ​​detection layer reads the latest standard physical examination image frame from each shared memory region according to the frame sequence number, discards the backlog of image frames with frame sequence numbers earlier than the latest standard physical examination image frame, and obtains the multimodal physical examination data to be detected.

3. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 1, characterized in that, Specifically, S3 is: Read the identity field from the identity data, perform text recognition on the surface image of the ID card to obtain the on-site identity field, and compare the identity field with the on-site identity field item by item to obtain the text verification result; Extract the on-site face region from continuous frontal human images, and input the identity portrait and the on-site face region into the DeepFace model to obtain the identity face features and the on-site face features respectively; The DeepFace model processes the identity face features and the on-site face features through a two-stage feature calibration layer. First, it performs overall structural calibration on the two types of face features to obtain the first-level calibration features. Then, it performs residual correction on the local differences in the first-level calibration features to obtain the second-level calibration features. Calculate the feature distance between the two types of secondary calibration features, and obtain the face comparison score based on the feature distance; Changes in eye state, head posture, and facial texture are extracted from continuous frontal human images. Liveness analysis results are obtained by comparing each change with the corresponding liveness threshold. The identity verification result is obtained based on the text verification result, the face comparison score, and the liveness analysis result.

4. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 1, characterized in that, The improved MediaPipe Pose model includes an image feature encoding module, a hierarchical pose recursion layer, a skeleton state reflow mechanism, and a key point decoding module. The image feature encoding module performs staged convolution processing on continuous human images to obtain pose feature maps corresponding to each image frame. The layered posture recursion layer divides the posture feature map into head and neck posture features, upper limb posture features, and trunk and lower limb posture features according to the connection relationship of the human skeleton. Within each type of posture feature, the key point state is recursively passed according to the key point connection order, and the various posture features are spliced ​​together to obtain the skeleton recursion feature. The skeleton state backflow mechanism extracts the deep key point states from the skeleton recursive features, returns the deep key point states to the corresponding preceding pose processing stage, and performs difference correction with the preceding key point states to obtain backflow correction features. The key point decoding module performs coordinate mapping on the reflow correction features to obtain the coordinates of human posture key points in each image frame, and arranges the human posture key point coordinates according to the acquisition order of the image frames to obtain continuous posture states.

5. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 4, characterized in that, The skeleton state reflow mechanism is specifically as follows: The deep key point states of each human body key point are extracted from the recursive features of the skeleton, and the correspondence between the deep key point states and the previous key point states is established according to the human body key point identifier. The state of each deep key point is passed to the corresponding preceding attitude processing stage along the return path opposite to the attitude recursion direction. Calculate the state difference between the state of the deep key point and the state of the corresponding preceding key point, and sort the key points of each human body according to the absolute value of the state difference to obtain the key point offset order. Human key points whose state difference exceeds the reflux threshold are identified as key points to be corrected. According to the key point offset order, the state of the corresponding deep key point is superimposed with the state of the previous key point to obtain the reflux state of the key point to be corrected. Based on the states of adjacent key points connected to the key point to be corrected, the return state is constrained by a skeleton to obtain the state of the corrected key point. The corrected key point state replaces the corresponding preceding key point state and is combined with the human body key point state that has not reached the backflow threshold to obtain the backflow correction feature.

6. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 1, characterized in that, Specifically, S5 is: Obtain the standard response information corresponding to each test item, and establish a response time window according to the execution order of the test items; The vision chart image is subjected to grayscale conversion and contour extraction, and the actual opening direction of the vision chart characters is determined based on the pixel distribution within the character area; The MediaPipe Hands model was used to extract the coordinates of key points of the hand from the frontal human image of the subject, and the direction of the subject's indication was determined based on the direction vector formed from the wrist key point to the fingertip key point. The visual response comparison result is obtained by comparing the direction indicated by the examinee with the actual opening direction of the characters on the visual chart. The test responses submitted by the test subjects within the corresponding response time windows of the colorimetry test and hearing test are obtained. Each test response is compared with the standard answer of the colorimetry chart and the pure tone playback channel to obtain the colorimetry response comparison results and the hearing response comparison results. The visual response comparison results, color recognition response comparison results, and hearing response comparison results are summarized according to the test item labels to obtain the response comparison results.

7. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 6, characterized in that, The process involves obtaining the test responses submitted by the subjects within the corresponding response time windows for colorimetry and hearing tests, comparing each response with the standard answer of the colorimetry chart and the pure tone playback channel, and obtaining the colorimetry response comparison results and hearing response comparison results. Extract colorimetric detection maps from the standard colorimetric map library and read the standard answers corresponding to the colorimetric detection maps; Extract the image features of the colorimetric spectrum image, compare the image features with the image features of each spectrum in the standard colorimetric spectrum library, and determine the actual colorimetric detection spectrum to be displayed based on the similarity ranking result; The test subject submits color discrimination answers through gesture or input operations within the corresponding response time window of the color discrimination ability test. The color discrimination answers are compared with the standard answers. The color discrimination response comparison results are obtained based on the number of correct responses within a preset number of questions. A sine wave audio signal containing fade-in and fade-out envelopes is generated through a pure tone generation program. The sine wave audio signal is played to the left and right channels respectively according to a preset frequency, and the pure tone playback channel, pure tone frequency and playback time are recorded. The hearing response submitted by the subject through gesture or input operation within the corresponding response time window of the hearing test is obtained, and the channel direction indicated by the hearing response is compared with the pure tone playback channel. The number of correct responses for the left and right channels at multiple frequencies was counted to obtain the hearing response comparison results.

8. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 1, characterized in that, Specifically, S6 is: According to the test item identifier, the identity verification result, the physical examination item feature set, and the response comparison result are respectively associated with the corresponding test item; The measurement values, action status, and duration of each test item are extracted from the feature set of the physical examination items, and invalid test data are filtered out based on the number of valid acquisition frames for each test item. Read the physical examination judgment threshold corresponding to each test item, and compare the measured value, the action state, the duration and the response comparison result with the corresponding physical examination judgment threshold to obtain the item judgment value of each test item; The face comparison score is compared with the identity verification threshold, and combined with the text verification result and the liveness analysis result to obtain the identity item judgment value; The model detection confidence value and response accuracy of each detection item are normalized according to the effective acquisition frame ratio, model detection confidence value and response accuracy, and the average value of the normalization results is calculated to obtain the item confidence of each detection item. By summarizing the item judgment value, the identity item judgment value, and the item confidence level according to the detection item identifier, the auxiliary detection result and the item confidence level are obtained.

9. The intelligent auxiliary detection method for driver physical examination based on multimodal AI according to claim 1, characterized in that, Specifically, S7 is: Read the system operating mode and determine the pure self-service testing process or the AI-assisted manual testing process based on the system operating mode; In the fully self-service testing process, the auxiliary testing results and confidence levels of the tested items are summarized according to the testing item identifiers to form the driver's physical examination auxiliary testing results; In the AI-assisted manual detection process, the auxiliary detection results of the item, the confidence level of the item, and the corresponding detection evidence are sent to the doctor's terminal; The manual review results and electronic confirmation information submitted by the doctor's terminal are obtained, and the manual review results are compared with the corresponding auxiliary test results item by item to obtain the consistency verification results. When the manual review result is consistent with the auxiliary test result of the project, the manual review result is used to form the auxiliary test result of the driver's physical examination; When the manual review result is inconsistent with the auxiliary test result of the item, the discrepancy test item, the auxiliary test result of the item, the manual review result and the reason for the discrepancy are recorded, and a special case mark is formed; The driver's physical examination auxiliary test results, the consistency verification results, and the test process data are associated with the examinee's identity identifier to form a driver's physical examination record.

10. A driver physical examination intelligent auxiliary detection system based on multimodal AI, comprising executing the driver physical examination intelligent auxiliary detection method based on multimodal AI as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The image acquisition module acquires the examinee's identity data, identity portrait, and ID card surface image through the identity reading device, and collects multimodal physical examination data through at least four universal RGB cameras of the same model; The preprocessing module establishes a fixed mapping between each general RGB camera and the detection item based on the camera device serial number and image acquisition capability identifier. It acquires each image through the C layer, performs format conversion and size adjustment on each image, and passes the latest image frame to the AI ​​detection layer through shared memory. The multi-model AI detection engine inputs the identity portrait and the on-site face into the DeepFace model, performs hierarchical extraction and difference calibration of facial features through a two-stage feature calibration layer, and obtains the identity verification result by combining text verification and liveness analysis; it inputs the human body image into the improved MediaPipe Pose model, recursively calculates the state of human body key points through a layered pose recursion layer, and corrects the state of cross-layer key points through a skeleton state backflow mechanism, and obtains the feature set of physical examination items by combining hand movements, head movements, vision and color discrimination analysis, and obtains the test subject's detection response in the corresponding test items; The hearing test module outputs a pure tone signal of a preset frequency through an audio device, obtains the test response submitted by the subject within the response time window, and compares the test response with the pure tone playback channel to obtain the hearing response comparison result. The fusion judgment engine integrates the identity verification results, the feature set of the physical examination items, and the response comparison results corresponding to each test item, and compares them with the physical examination judgment threshold to obtain the auxiliary test results and the confidence level of the items. The dual-track output module generates driver physical examination auxiliary test results based on the auxiliary test results of the project in the pure self-service mode, and sends the auxiliary test results of the project, the confidence level of the project and the test evidence to the doctor's terminal in the AI-assisted human mode to obtain the manual review results and perform consistency verification. The cloud-based monitoring platform performs encrypted transmission, resume interrupted transmission, remote configuration management, and full-process log recording on the driver's physical examination auxiliary detection results and detection process data.

Images