Computer-aided surface access method and system based on Web technology and deep neural network

Abstract

The invention discloses a computer-aided face visiting method and system based on a Web technology and a deep neural network, and the method comprises the steps: an accessor fills in a questionnaire during face visiting, and uploads the questionnaire data to a cloud end in real time; performing basic verification, abnormal value detection and logical relationship check on the text data in sequence to obtain qualified text data; performing verification according to the attribute of the audio, marking abnormal data in the verification process, and giving a prompt to an accessor; performing voice separation on the verified audio, and counting the number of sound sources; comparing the number of the sound sources with a preset number, and marking whether the data is valid or not; recognizing human faces and counting the number of human faces in the image data, then performing similarity judgment on human face features, and reserving the image data meeting set conditions; and checking the positioning data, and marking whether the data is valid or not. By automatically and intelligently checking the questionnaire information, the efficiency and accuracy of questionnaire processing and statistics are improved, and the data quality is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of information processing, and in particular relates to a computer-assisted face-to-face interview method and system based on Web technology and deep neural network. Background Art

[0002] The face-to-face interview method of social survey has undergone many evolutions, from the initial paper questionnaire to the electronic questionnaire on the PC, and then to the electronic questionnaire on mobile terminals such as mobile phones and tablets, spanning three stages of development. At present, most computer-assisted face-to-face interview systems are built based on App client technology, and during the interview process, there is no compliance and availability verification of the recording file, such as whether there is sound, whether the sound is clear, and whether it contains multiple voices; when uploading the interview picture, there is no legality verification of whether the picture is a photo of multiple people; and after the questionnaire is completed, there is no integrity and validity verification of the questionnaire data; this leads to a series of problems such as high development and maintenance costs, inconvenient system updates, delayed data transmission, and the inability to determine the integrity, legality and availability of questionnaire data, recording files and interview pictures. Summary of the invention

[0003] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a computer-assisted face-to-face interview method and system based on Web technology and deep neural network, which improves the efficiency and accuracy of questionnaire processing and statistics by automatically verifying questionnaire information, and solves the problems of data lag and data security by adopting cloud storage technology and page-segmented recording technology.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a computer-assisted face-to-face interview method based on Web technology and deep neural network, comprising the following steps: During the face-to-face interview, the interviewer fills in the questionnaire and uploads the questionnaire data to the data storage module in real time. The questionnaire data includes text data, audio data, image data and positioning data. Read the text questionnaire data, perform basic verification on the text data using predefined basic verification rules, and use the model trained based on historical questionnaire filling data to perform outlier detection and logical relationship check on the values ​​of each field in the questionnaire data to obtain qualified text data; Read audio data and verify it according to the audio attributes. During the verification process, mark the audio that meets the conditions as abnormal and give a prompt to the interviewer; perform voice separation on the verified audio and count the number of sound sources; compare the number of sound sources with the preset number to check the consistency. If there is inconsistency, prompt the interviewer and mark the audio status as abnormal; Read image data, identify faces and count the number of faces in the image data, retain image data whose number of faces meets the set conditions, extract facial features from the image data, perform similarity judgment on the facial features, and retain image data that meets the set conditions; Verify the positioning data and mark the questionnaire data within the set range as valid, and those outside the range as invalid.

[0005] As a preferred technical solution, the user fills in the questionnaire and uploads the questionnaire data to the data storage module in real time, including: The electronic questionnaire is set up according to the needs by using the questionnaire engine; the electronic questionnaire includes a text input module, a recording module, a camera module and a positioning module, the text input module is used to provide a space for users to input questionnaire answer data, the recording module is used to provide a shortcut key or link to connect a cross-platform recording application to the electronic questionnaire, the camera module is used to provide a shortcut key or link to connect a cross-platform camera application to the electronic questionnaire, and the positioning module is used to provide a shortcut key or link to connect a cross-platform navigation or positioning application to the electronic questionnaire; After the electronic questionnaire is delivered, the user enters data according to the questions in the electronic questionnaire, specifically: Enter text content according to the question and obtain text data; The audio content is recorded in a segmented manner. When the questionnaire switches interfaces, the audio recording is interrupted. When a new interface appears, the audio recording continues until all current interviews are completed. The audio recording modes include partial recording, full recording, and no recording. Shooting or directly transmitting and uploading image data through a camera module, the image data includes static pictures, dynamic images and videos; Record the current interviewee's location coordinates or network coordinates through the positioning module; After the collection is completed, the questionnaire data will be synchronized to the cloud.

[0006] As a preferred technical solution, the basic verification of text data using basic verification rules is specifically as follows: Set the numerical range of questionnaire attributes, compare the text data with the numerical values ​​corresponding to the questionnaire attributes, and when the numerical values ​​match, the text data verification passes the basic verification.

[0007] As a preferred technical solution, the numerical features of each field in the text data that has completed the basic verification are extracted, specifically: The isolation forest model is trained by using historical data and extracting numerical features of each field in these questionnaires; The trained isolation forest model is used to extract numerical features from the text data that has completed the basic test, obtain various numerical features, and test each numerical feature. The numerical features include basic user information, a certain type of classified questions, and answer keywords.

[0008] As a preferred technical solution, the logic relationship verification is specifically as follows: Use the logistic regression model to detect the logical relationship between numerical features. If there is a logical relationship between the numerical features, the test passes. Otherwise, the user is prompted to re-fill the form. The logical relationships include relationships determined by natural laws, relationships stipulated by laws, and user-defined relationships.

[0009] As a preferred technical solution, the verification is performed according to the attributes of the audio, and during the verification process, the audio that meets the conditions is marked as abnormal and a prompt is given to the interviewer, specifically: Analyze the volume level and energy value of the audio, analyze the volume of each audio sequence according to the set threshold, mark the audio status as abnormal when the volume is less than the set threshold and give a prompt to the interviewer, use the signal-to-noise ratio to analyze the clarity of the remaining audio sequences, mark the audio status as abnormal when the clarity is less than the set threshold and give a prompt to the interviewer.

[0010] As a preferred technical solution, the clear audio sequence is separated into voices and the number of sound sources is counted, specifically: Using a deep learning library to identify the source of sound in a sequence, extracting sound features from an audio signal, and dividing the audio into several segments based on the sound features, each segment of audio only contains the voice of a single speaker; the sound features include Mel frequency cepstral coefficients, logarithmic Mel frequency spectrum, and acoustic features; When extracting sound features, the silent segments in the audio are used as segmentation points, and a clustering algorithm is used to assign different segments to different speakers, and the number of sound sources is counted.

[0011] As a preferred technical solution, the method of identifying faces and counting the number of faces in the image data, and retaining the image data whose number of faces meets the set conditions, is specifically: Use the MTCNN algorithm to determine whether there is a face in the image, and output the image with a face to the next step; The number of faces in the image is counted to determine whether the image is a group photo. If the number of faces exceeds two, the current face image is saved and the number of faces is marked. Otherwise, the user is prompted to re-upload the group photo.

[0012] As a preferred technical solution, the extraction of facial features from image data, similarity judgment of the facial features, and retention of image data that meets the set conditions are specifically as follows: Assuming that there are two faces, the facial feature vector is extracted through the FaceNet model, and the similarity between the two facial features is calculated through the Euclidean distance algorithm or cosine similarity, and it is determined whether the faces in the image are the same person. If it is determined to be the same person, the user is prompted to upload the image again, otherwise the image is saved.

[0013] In a second aspect, the present invention further provides a computer-assisted face-to-face interview system based on Web technology and deep neural network, which is applied to the computer-assisted face-to-face interview method based on Web technology and deep neural network, including a questionnaire collection module, a data storage module, a text verification module, an audio verification module, an image verification module, a positioning verification module and a control display module; The questionnaire collection module is used for the interviewer to fill in the questionnaire during the face-to-face interview and upload the questionnaire data to the data storage module in real time. The questionnaire data includes text data, audio data, image data and positioning data; A data storage module, used to upload data to the cloud and provide data to the text verification module, the audio verification module, the image verification module and the positioning verification module; The text verification module reads the text questionnaire data, performs basic verification on the text data using predefined basic verification rules, and uses a model trained based on historical questionnaire filling data to perform abnormal value detection and logical relationship check on the values ​​of each field in the questionnaire data to obtain qualified text data; The audio verification module is used to verify the audio according to its properties. During the verification process, the audio that meets the conditions will be marked as abnormal and the interviewer will be prompted. The verified audio will be separated into voices and the number of sound sources will be counted. The number of sound sources will be compared with the preset number to check the consistency. If there is any inconsistency, the interviewer will be prompted and the audio status will be marked as abnormal. The image verification module is used to read the image data, identify the face and count the number of faces in the image data, retain the image data whose number of faces meets the set conditions, extract the face features in the image data, perform similarity judgment on the face features, and retain the image data that meets the set conditions; The positioning verification module is used to verify the positioning data, marking the questionnaire data within the set range as valid and those outside the range as invalid; The control display module is used to operate and count various instructions and visualize various output results.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The present invention improves the efficiency and accuracy of questionnaire processing and statistics by automatically verifying questionnaire information. At the same time, by real-time verification of questionnaire data such as text data, audio data, image data, and positioning data, the integrity, rationality, and effectiveness of questionnaire data can be ensured.

[0015] (2) The present invention designs an electronic questionnaire through a questionnaire engine and connects cross-platform applications through various modules. When using the electronic questionnaire, there is no need to download the relevant applications and the functions of the relevant cross-platform applications can be used, which not only reduces the development and maintenance costs but also improves the user experience.

[0016] (3) The present invention adopts cloud storage technology and page-segmented recording technology. When the next page button of the questionnaire is clicked, the recording is cut off, and then the recording file and questionnaire-related data are transmitted back to the cloud in real time, thereby solving the problems of data lag and data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flow chart of a computer-assisted interview method based on Web technology and deep neural network according to an embodiment of the present invention; Figure 2 A flowchart of text data verification according to an embodiment of the present invention; Figure 3 A flowchart of audio data verification according to an embodiment of the present invention; Figure 4 A flowchart of image data verification according to an embodiment of the present invention; Figure 5 It is a structural schematic diagram of a computer-assisted face-to-face interview system based on Web technology and deep neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0020] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0021] See also Figure 1 This embodiment provides a computer-assisted face-to-face interview method based on Web technology and deep neural network, including the following steps: S1. The user fills in a questionnaire and uploads the questionnaire data to a data storage module in real time. The questionnaire data includes text data, audio data, image data, and positioning data.

[0022] The audio, image and positioning data collected in this embodiment are all used for interview quality control to ensure that the data is authentic and valid, and that the data is indeed the questionnaire completed by the interviewee through face-to-face interview, rather than invalid data filled in by the interviewer himself.

[0023] In order to better collect these questionnaire data, in this embodiment, the developer uses the questionnaire engine to set up an electronic questionnaire as needed. The electronic questionnaire is provided with multiple modules, including a text input module, a recording module, a camera module, and a positioning module, etc. These modules can be designed and used according to actual needs. The questionnaire engine module, the recording module, the camera module, and the positioning module are all implemented based on Web technology, wherein the questionnaire engine is developed based on the uni-app and vue mobile web page frameworks, the recording is implemented using a third-party web page recording component, the image module is implemented based on the h5 file, and the positioning is implemented based on the Baidu map WebAPI. In addition, the browser compatibility adaptation of various mobile terminals is done, and the system can run on various mobile terminals, realizing a set of systems adapted to multiple terminals. These modules can provide shortcut keys or links to connect the corresponding applications to the electronic questionnaire. In the absence of various applications, users can directly call the application functions through shortcut keys or links combined with the current hardware device components. In addition, in addition to the above-mentioned text input, recording, video recording, positioning and other applications, this embodiment can also realize cross-platform applications such as text symbol format conversion, language text conversion, automatic text correction, image editing, speech to text, multi-language recognition, and face AI recognition under a set of system architecture based on Web technology.

[0024] Next, after the developer posts the electronic questionnaire, the user enters data according to the questions in the electronic questionnaire, specifically: S11. Input text content according to the question and obtain text data.

[0025] S12. Record the audio content in a segmented manner. When the questionnaire switches interfaces, the audio recording is interrupted. When a new interface appears, the audio recording is continued until all current interviews are completed. The audio recording modes include partial recording, full recording, and no recording.

[0026] Furthermore, step S12 can also allow the user to select a recording mode as needed according to project requirements, and the recording mode is set at the place where the questionnaire is written, and the system will perform the corresponding recording function according to the mode set by the user.

[0027] Regardless of the recording mode, the recording method is implemented by recording in segments by page. The specific implementation is: after entering the questionnaire, the recording permission will be obtained first. If it is a full recording, the whole process will be recorded. When the user clicks the next page, the recording will be turned off, and then the recording, questionnaire data, and parallel data will be uploaded to the cloud. After entering the next page, the recording will be reopened, and so on. If it is a partial recording, when the questions that need to be recorded appear according to the questionnaire settings, the recording will be automatically triggered. When the next page is clicked, the recording will be uploaded to the cloud together with the questionnaire data, and the others are similar. In this way, real-time uploading of recordings and questionnaire data can be achieved.

[0028] S13. Shooting image data through a camera module or directly transmitting and uploading image data, wherein the image data includes static images, dynamic images and videos.

[0029] S14. Record the location coordinates or network coordinates of the current interviewee through the positioning module.

[0030] S15. After the collection is completed, the questionnaire data is synchronized to the cloud.

[0031] In this embodiment, the data storage method is implemented by cloud storage and page upload. When the user clicks the next page, if the recording is in progress, the recording will be closed first, and then the recording file will be read into the memory. The recording, questionnaire data and parallel data will be sent to the server through a post request for storage. The recording file is stored in the DFS file system, and the data is stored in the mongodb and mysql databases. This not only solves the problem of delayed recording data, but also adds paging or segmented recording, which is convenient for improving the efficiency of quality control verification.

[0032] The above step S1 is the questionnaire collection part of this embodiment. For this embodiment, the focus is on real-time verification of the integrity, rationality and validity of the questionnaire data, for which this embodiment provides steps S2-S5.

[0033] S2. Read the text data, perform basic verification on the text data using basic verification rules, extract the numerical features of each field in the text data that has completed the basic verification, perform logical relationship verification on the numerical features, and obtain qualified text data.

[0034] In step S2, the text data needs to be checked three times, namely, basic check, rationality check and logical relationship check. Figure 2 shown.

[0035] S21, basic verification. Set the numerical range of the questionnaire attribute, compare the text data with the numerical value corresponding to the questionnaire attribute, and when the numerical values ​​match, the text data verification passes the basic verification.

[0036] For example, the range of values ​​for each question in the questionnaire, the number of options, etc., and verify the data and questions.

[0037] S22. Reasonableness check. Use historical data and extract the numerical features of each field in these questionnaires to train the Isolation Forest model; use the trained Isolation Forest model to extract numerical features of the text data that has completed the basic test, obtain various numerical features and then detect the abnormal values ​​of these features. The numerical features include basic user information, a certain type of classified questions and answer keywords.

[0038] Among them, the basic information of the user can be name, social relationship, height, income, age, etc.; the answer keywords are some verbs, nouns, etc. with positions or emotions to answer the questionnaire questions, such as "identify", "agree", "dislike", etc.

[0039] S23, logical relationship verification. A binary classification model, namely a logistic regression model, is used to detect the logical relationship between numerical features. If there is a logical relationship between the numerical features, the verification passes. Otherwise, a prompt is given to the user to re-fill the form. The logical relationship includes a relationship determined by natural laws, a relationship stipulated by laws, and a custom relationship.

[0040] For example, using "age" and "marital status" as features, the model is trained to predict the rationality of the questionnaire data. If the value of "age" (for example, 16 years old) and "marital status" is "married", it should be considered unreasonable.

[0041] S3. Read the audio data, cut the audio sequence, analyze the volume of each audio sequence according to the set threshold, delete the audio sequences with volume less than the set threshold, use the signal-to-noise ratio to perform clarity analysis on the remaining audio sequences, separate the clear audio sequences from speech, count the number of sound sources, compare the number of sound sources in the actual interview with the counted number, and verify consistency.

[0042] like Figure 3 As shown, the verification of audio data is performed using the following steps: S31. Determine whether the recording file contains sound by analyzing the volume level and energy value of the audio. Specific implementation method: Use the pydub library to load the audio file, split the audio into short frames, and calculate the volume level of each frame. Determine whether the volume reaches the set threshold. If the volume is low, it can be considered that there is no valid sound in the audio.

[0043] S32. Determine whether the recording is clear by analyzing the signal-to-noise ratio (SNR) of the audio. Specific implementation method: Use the librosa library to analyze the energy of the signal to estimate the signal-to-noise ratio. Set a threshold to determine whether the recording is clear.

[0044] S33, using the deep learning library to identify the sound source in the sequence, and using the speech separation technology (SpeakerDiarization) to separate the clear audio sequence according to the characteristics of each sound source. Specifically, extract the sound features from the audio signal, the sound features include MFCC (Mel Frequency Cepstrum Coefficient), Log Mel Spectrogram and acoustic features, etc., divide the audio into several sections according to these sound features, each section of audio has only the voice of a single speaker, and separate the clear audio sequence according to the characteristics of each sound source through the speech separation technology, obtain the separated audio sequence, count the number of separated audio sequences, and obtain the number of sound sources. Divide the audio into several sections to ensure that each section contains the voice of a single speaker, where the segmentation method includes: a fixed window based on time (such as one section every 2 seconds) and dynamic segmentation: using the silent section in the audio as the segmentation point. Use the clustering algorithm to assign different sections to different speakers, and count how many speakers there are to count the number of sound sources. It should be noted that the speech separation technology used in this embodiment supports "K-means clustering". In general interviews, the deep clustering model is trained using the PyAudioAnalysis deep learning library to determine how many sub-sequences are synthesized in the audio sequence. Once the number of these sub-sequences is obtained, the number of sound sources can be known. In addition, in order to eliminate environmental noise in the recording, it is also necessary to compare the number of faces in the subsequent images to further confirm the consistency between the interviewer and the sound source.

[0045] S4, reading the image data, identifying faces and counting the number of faces in the image data, retaining the image data whose number of faces meets the set conditions, extracting facial features in the image data, performing similarity judgment on the facial features, and retaining the image data that meets the set conditions.

[0046] like Figure 4As shown in the figure, the MTCNN algorithm is used to determine whether there is a face in the picture, and the number of faces is counted to determine whether the picture is a group photo. Specific implementation method: Use the output of the face detection model to calculate the number of face areas and ensure that the number is equal to more than two. For example: mark the detected faces and calculate the number of face frames. If the number of detected faces is not greater than or equal to 2, prompt the user to upload a group photo containing more than two people.

[0047] The 128-dimensional feature vector of the face is extracted through the FaceNet model, and then the similarity of the two facial features is calculated through the Euclidean distance algorithm to determine whether the picture is composed of the same person. In order to ensure that multiple faces belong to different people (to prevent users from uploading pictures composed of multiple photos of the same person), the similarity can be compared through the facial feature vector. Specific implementation method: Extract the 128-dimensional feature vector of the face through the FaceNet model. Then calculate the similarity of the two facial features through the Euclidean distance. If the similarity between the two faces is high, it can be judged that they are the same person. If the conditions are not met, it is required to re-upload the picture.

[0048] S5 is used to verify the positioning data, marking the questionnaire data within the set range as valid, and marking the questionnaire data outside the range as invalid.

[0049] Set a certain geographic or domain targeting range, frame the electronic questionnaire within this range, and exclude those outside the other ranges. This is to avoid data errors during the survey.

[0050] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0051] Based on the same idea as the computer-assisted face-to-face interview method based on Web technology and deep neural network in the above-mentioned embodiment, the present invention also provides a computer-assisted face-to-face interview system based on Web technology and deep neural network, which can be used to execute the above-mentioned computer-assisted face-to-face interview method based on Web technology and deep neural network. For ease of explanation, in the structural schematic diagram of the embodiment of the computer-assisted face-to-face interview system based on Web technology and deep neural network, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0052] See also Figure 5In another embodiment of the present application, a computer-assisted face-to-face interview system 10 based on Web technology and deep neural network is provided, the system comprising a questionnaire collection module 11, a data storage module 12, a text verification module 13, an audio verification module 14, an image verification module 15, a positioning verification module 16 and a control display module 17; Questionnaire collection module 11, used for interviewers to fill in questionnaires during face-to-face interviews and upload questionnaire data to the data storage module in real time, wherein the questionnaire data includes text data, audio data, image data and positioning data; The data storage module 12 is used to upload data to the cloud and provide data to the text verification module, the audio verification module, the image verification module and the positioning verification module; The text verification module 13 reads the text questionnaire data, performs basic verification on the text data using predefined basic verification rules, and uses a model trained based on historical questionnaire filling data to perform abnormal value detection and logical relationship check on the values ​​of each field in the questionnaire data to obtain qualified text data; The audio verification module 14 is used to verify the audio according to its properties. During the verification process, the audio that meets the conditions is marked as abnormal and a prompt is given to the interviewer; the audio that has completed the verification is separated into voices and the number of sound sources is counted; the number of sound sources is compared with the preset number to check the consistency. If there is inconsistency, the interviewer is prompted and the audio status is marked as abnormal; The image verification module 15 is used to read the image data, identify the human face and count the number of human faces in the image data, retain the image data whose number of human faces meets the set conditions, extract the facial features in the image data, perform similarity judgment on the facial features, and retain the image data that meets the set conditions; The positioning verification module 16 is used to verify the positioning data, marking the questionnaire data within the set range as valid, and marking the questionnaire data outside the range as invalid; The control display module 17 is used to operate and count various instructions and visualize various output results.

[0053] It should be noted that the computer-assisted interview system based on Web technology and deep neural network of the present invention corresponds one-to-one to the computer-assisted interview method based on Web technology and deep neural network of the present invention. The technical features and beneficial effects described in the embodiment of the computer-assisted interview method based on Web technology and deep neural network are applicable to the embodiment of the computer-assisted interview method based on Web technology and deep neural network. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0054] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0055] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A computer-assisted interview method based on Web technology and deep neural network, characterized in that: The steps include: During the face-to-face interview, the interviewer fills in the questionnaire and uploads the questionnaire data to the data storage module in real time. The questionnaire data includes text data, audio data, image data and positioning data. Read the text questionnaire data, perform basic verification on the text data using predefined basic verification rules, and use the model trained based on historical questionnaire filling data to perform outlier detection and logical relationship check on the values ​​of each field in the questionnaire data to obtain qualified text data; Read the audio data and verify it according to the audio attributes. During the verification process, mark the audio that meets the conditions as abnormal and give a prompt to the interviewer; Perform voice separation on the verified audio and count the number of sound sources; compare the number of sound sources with the preset number to check consistency. If there is any inconsistency, the interviewer will be notified and the audio status will be marked as abnormal; Read image data, identify faces and count the number of faces in the image data, retain image data whose number of faces meets the set conditions, extract facial features from the image data, perform similarity judgment on the facial features, and retain image data that meets the set conditions; Verify the positioning data and mark the questionnaire data within the set range as valid, and those outside the range as invalid.

2. According to claim 1, the computer-assisted interview method based on Web technology and deep neural network is characterized in that: The user fills in the questionnaire and uploads the questionnaire data to the data storage module in real time, including: The electronic questionnaire is set up according to the needs by using the questionnaire engine; the electronic questionnaire includes a text input module, a recording module, a camera module and a positioning module, the text input module is used to provide a space for users to input questionnaire answer data, the recording module is used to provide a shortcut key or link to connect a cross-platform recording application to the electronic questionnaire, the camera module is used to provide a shortcut key or link to connect a cross-platform camera application to the electronic questionnaire, and the positioning module is used to provide a shortcut key or link to connect a cross-platform navigation or positioning application to the electronic questionnaire; After the electronic questionnaire is delivered, the user enters data according to the questions in the electronic questionnaire, specifically: Enter text content according to the question and obtain text data; The audio content is recorded in a segmented manner. When the questionnaire switches interfaces, the audio recording is interrupted. When a new interface appears, the audio recording continues until all current interviews are completed. The audio recording modes include partial recording, full recording, and no recording. Shooting or directly transmitting and uploading image data through a camera module, the image data includes static pictures, dynamic images and videos; Record the current interviewee's location coordinates or network coordinates through the positioning module; After the collection is completed, the questionnaire data will be synchronized to the cloud.

3. According to claim 1, the computer-assisted interview method based on Web technology and deep neural network is characterized in that: The basic verification of text data using basic verification rules is specifically as follows: Set the numerical range of questionnaire attributes, compare the text data with the numerical values ​​corresponding to the questionnaire attributes, and when the numerical values ​​match, the text data verification passes the basic verification.

4. According to claim 1, the computer-assisted interview method based on Web technology and deep neural network is characterized in that: The numerical features of each field in the text data that has completed the basic verification are extracted, specifically: The isolation forest model is trained by using historical data and extracting numerical features of each field in these questionnaires; The trained isolation forest model is used to extract numerical features from the text data that has completed the basic test, obtain various numerical features, and test each numerical feature. The numerical features include basic user information, a certain type of classified questions, and answer keywords.

5. According to claim 1, the computer-assisted interview method based on Web technology and deep neural network is characterized in that: The logic relationship verification is specifically as follows: Use the logistic regression model to detect the logical relationship between numerical features. If there is a logical relationship between the numerical features, the test passes. Otherwise, the user is prompted to re-fill the form. The logical relationships include relationships determined by natural laws, relationships stipulated by laws, and user-defined relationships.

6. The computer-assisted interview method based on Web technology and deep neural network according to claim 1 is characterized in that: The verification is performed according to the properties of the audio, and during the verification process, the audio that meets the conditions is marked as abnormal and a prompt is given to the interviewer, specifically: Analyze the volume level and energy value of the audio, analyze the volume of each audio sequence according to the set threshold, mark the audio status as abnormal when the volume is less than the set threshold and give a prompt to the interviewer, use the signal-to-noise ratio to analyze the clarity of the remaining audio sequences, mark the audio status as abnormal when the clarity is less than the set threshold and give a prompt to the interviewer.

7. The computer-assisted interview method based on Web technology and deep neural network according to claim 6 is characterized in that: The clear audio sequence is separated into voices and the number of sound sources is counted, specifically: Using a deep learning library to identify the source of sound in a sequence, extracting sound features from an audio signal, and dividing the audio into several segments based on the sound features, each segment of audio only contains the voice of a single speaker; the sound features include Mel frequency cepstral coefficients, logarithmic Mel frequency spectrum, and acoustic features; When extracting sound features, the silent segments in the audio are used as segmentation points, and a clustering algorithm is used to assign different segments to different speakers, and the number of sound sources is counted.

8. The computer-assisted interview method based on Web technology and deep neural network according to claim 1 is characterized in that: The method of identifying faces and counting the number of faces in the image data, and retaining the image data whose number of faces meets the set conditions, is specifically: Use the MTCNN algorithm to determine whether there is a face in the image, and output the image with a face to the next step; The number of faces in the image is counted to determine whether the image is a group photo. If the number of faces exceeds two, the current face image is saved and the number of faces is marked. Otherwise, the user is prompted to re-upload the group photo.

9. The computer-assisted interview method based on Web technology and deep neural network according to claim 1, characterized in that: The extraction of facial features from the image data, the similarity judgment of the facial features, and the retention of the image data that meets the set conditions are specifically as follows: Assuming that there are two faces, the facial feature vector is extracted through the FaceNet model, and the similarity between the two facial features is calculated through the Euclidean distance algorithm or cosine similarity, and it is determined whether the faces in the image are the same person. If it is determined to be the same person, the user is prompted to upload the image again, otherwise the image is saved.

10. A computer-assisted interview system based on Web technology and deep neural network, characterized in that: A computer-assisted face-to-face interview method based on Web technology and deep neural network applied to any one of claims 1 to 9, comprising a questionnaire collection module, a data storage module, a text verification module, an audio verification module, an image verification module, a positioning verification module and a control display module; The questionnaire collection module is used for the interviewer to fill in the questionnaire during the face-to-face interview and upload the questionnaire data to the data storage module in real time. The questionnaire data includes text data, audio data, image data and positioning data; A data storage module, used to upload data to the cloud and provide data to the text verification module, the audio verification module, the image verification module and the positioning verification module; The text verification module reads the text questionnaire data, performs basic verification on the text data using predefined basic verification rules, and uses a model trained based on historical questionnaire filling data to perform abnormal value detection and logical relationship check on the values ​​of each field in the questionnaire data to obtain qualified text data; The audio verification module is used to verify the audio according to its attributes. During the verification process, the audio that meets the conditions is marked as abnormal and a prompt is given to the interviewer. Perform voice separation on the verified audio and count the number of sound sources; compare the number of sound sources with the preset number to check consistency. If there is any inconsistency, the interviewer will be notified and the audio status will be marked as abnormal; The image verification module is used to read the image data, identify the face and count the number of faces in the image data, retain the image data whose number of faces meets the set conditions, extract the face features in the image data, perform similarity judgment on the face features, and retain the image data that meets the set conditions; The positioning verification module is used to verify the positioning data, marking the questionnaire data within the set range as valid and those outside the range as invalid; The control display module is used to operate and count various instructions and visualize various output results.

Images