A multi-modal data processing method and system based on user investigation

Abstract

The application discloses a kind of based on user investigation multi-modal data processing method and system, method includes: creating user investigation session and maintaining context index;During investigation execution, in response to event, write event mark carrying time offset and context index without interrupting collection;After collection, according to event mark, multi-modal data is divided, and data segment is bound with context index;Above context index is unit, transcribed text, stimulus material, user feedback and external verification product are aggregated, structured evidence line is generated, and multi-stage analysis including social influence audit and evidence integrity calculation is executed based on this;According to the association of analysis conclusion and evidence line, the investigation report containing evidence traceability is output.This application also supports offline collection, result snapshot upload, asynchronous collaboration and safe sharing.This application can realize the automation of user investigation data, evidence chain is traceable and the reliability of investigation conclusion is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal data processing method and system based on user surveys. Background Technology

[0002] User research is a core method for obtaining user feedback in product design, market research, and user experience optimization. A typical user research process usually includes multiple stages such as research preparation, on-site execution, data collection (audio / video recording), sample presentation, participant rating, material collection, data transcription, analysis and research, and report output. This process generates a large amount of diverse data, including but not limited to audio, video, text, images, rating forms, user assignments, and outputs generated by external tools.

[0003] Currently, various technological solutions exist in the industry to assist or automate certain aspects of user research, such as interview text post-processing and report generation technologies, semantic analysis-based research report generation technologies, online interview communication and text processing technologies, meeting document generation technologies, and user profiling technologies. However, these existing technologies generally suffer from the following shortcomings and deficiencies when dealing with complex user research scenarios:

[0004] (1) Most existing technologies are centered on a single data form (such as interview text, conference subtitles) or a single result (such as user profile). There is a lack of a unified context index established during the research operation period, and the system cannot reliably and accurately associate a certain audio segment, a certain rating or a certain product version with the specific research question, participants and research unit to which it belongs in a timely manner.

[0005] (2) Because the on-site data was not structured and correlated, researchers had to spend a lot of time manually aligning it after the survey. For example, they had to compare the recording timeline, survey outline, rating sheet and notes to reconstruct "which participant gave what feedback for which sample". This kind of manual operation after the fact is not only inefficient, but also very prone to error.

[0006] (3) Existing report generation solutions mostly rely on summarizing or mapping the entire interview text into a knowledge graph. There is a lack of machine-readable, structured "evidence line" links. When it is necessary to backtrack and verify a specific conclusion in the report, manual searching in massive amounts of data is still required.

[0007] Therefore, there is an urgent need for a system and method that can uniformly manage, correlate in real time, and organize multimodal data throughout the entire user survey process in a structured manner, and generate traceable, verifiable, and interference-resistant survey conclusions based on this data. Summary of the Invention

[0008] To address the aforementioned issues, this application discloses a multimodal data processing method and system based on user surveys, enabling the collection, storage, segmentation, analysis, output, and verification of audio recordings, videos, texts, samples, ratings, votes, external validation products, user tasks, and report conclusions generated during the survey operation period, all within the same context index.

[0009] Technical solution:

[0010] A multimodal data processing method based on user surveys, characterized by the following steps:

[0011] S1, create a user survey session and maintain a runtime context index including the current survey unit identifier, the current question identifier, and the current participant identifier;

[0012] S2, during the survey execution process, in response to runtime events, an event marker is written on the acquisition timeline without interrupting multimodal data acquisition. The event marker carries a time offset and the runtime context index.

[0013] S3. After the data collection is completed, the collected multimodal data is segmented according to the event markers, and the segmented data fragments are bound to the corresponding runtime context indexes to generate transcribed text.

[0014] S4, using the runtime context index as a unit, aggregate the transcribed text, corresponding stimulus materials, user feedback data, and external verification products to generate a structured evidence row;

[0015] S5. Based on the structured evidence row, perform multi-stage analysis, which includes social impact audit and evidence completeness calculation, and generate analysis conclusions with evidence completeness markers.

[0016] S6. Based on the correlation between the analysis conclusions and the structured evidence rows, perform report generation and self-checking of evidence constraints, and output a research report containing evidence tracing.

[0017] Furthermore, in step S2, the runtime events include at least one of the following: survey unit switching, question switching, participant switching, scoring, voting, external editing start, and external editing save.

[0018] Furthermore, in step S3, the binding includes: determining the start and end times of the audio / video segment based on the time offset between two adjacent event markers; performing speech recognition on the audio / video segment to generate transcribed text; and associating the transcribed text with the runtime context index in the event marker corresponding to the start time.

[0019] Furthermore, in step S4, the structured evidence row includes at least: survey unit information, question information, participant identifiers, transcribed text, stimulus material identifiers and metadata, rating and voting matrix, external product version records, and event tag list.

[0020] Furthermore, in step S5, the social impact audit includes: analyzing the speaking sequence and semantic relevance of different participants on the same issue, identifying echoing, soft opposition, moderator guidance, and assertive speaking behaviors; and downgrading or labeling the evidence completeness of conclusions affected by the above behaviors.

[0021] Furthermore, the completeness of evidence is calculated using the following formula:

[0022] ,

[0023] In the formula, , , , These are, respectively, the completeness of the transcribed text, the completeness of the quantitative feedback, the completeness of the evidence object, and the completeness of the analysis results; , , , The corresponding weights.

[0024] Furthermore, in step S6, the report generation and self-checking of the evidence constraints include: classifying the analysis conclusions into consensus, majority opinion, individual opinion, and opinion to be verified based on the evidence completeness markers and social impact audit results; scanning the generated report text to detect conclusions lacking evidence support, internal contradictions, and inconsistencies between user verbal expressions and ratings, and outputting the self-checking results.

[0025] Furthermore, the method also includes: organizing the imported multiple stimulus materials into at least one survey unit; when adding, deleting or regrouping the stimulus materials, migrating the associated data in the original survey unit to the new survey unit according to the overlapping relationship of the material identifiers in the old and new survey units.

[0026] Furthermore, user feedback data includes ratings and / or votes, and is stored as a composite key using a combination of the runtime context index and the evidence object identifier.

[0027] Furthermore, the external validation product is generated in the following manner: in response to an external processing request, a working copy of the original stimulus material is created; an external processing tool is invoked to process the working copy and the processing parameters are recorded; the processed final version is imported and user feedback on the final version is recorded.

[0028] Furthermore, the method also includes: in a multi-person survey scenario, assigning audio segments to users based on participant switching markers; when multiple participants are detected but lack participant switching markers, marking the corresponding audio segments as pending assignment, and providing prompts or calling speaker separation services for auxiliary assignment in subsequent processing.

[0029] This invention also discloses a multimodal data processing system based on user surveys, comprising:

[0030] The survey session management module is used to create and maintain user survey sessions, as well as a runtime context index consisting of the current survey unit identifier, the current question identifier, and the current participant identifier.

[0031] The event tagging module is used to write event tags carrying time offsets and the runtime context index on the acquisition timeline in response to runtime events during the survey execution process, without interrupting multimodal data acquisition.

[0032] The multimodal preprocessing module is used to segment the multimodal data according to the event markers after the acquisition is completed, and to transcribe the segmented data fragments and bind the transcribed text to the corresponding runtime context index.

[0033] The structured corpus construction module is used to aggregate transcribed text, stimulus materials, user feedback data, and external validation products into structured evidence rows, using the runtime context index as the unit.

[0034] The hierarchical analysis module is used to perform multi-stage analysis based on the structured evidence line, the multi-stage analysis including social impact audit and evidence completeness calculation, and generate analysis conclusions with evidence completeness markers;

[0035] The report generation module is used to perform report generation and self-checking based on the correlation between the analysis conclusions and the structured evidence rows, and output a research report containing evidence tracing.

[0036] Furthermore, the event tagging module supports runtime events including at least one of the following: survey unit switching, question switching, participant switching, rating, voting, external editing start, and external editing save.

[0037] Furthermore, the hierarchical analysis module is also used to: analyze the speaking time sequence and semantic relevance of different participants on the same issue, identify echoing, soft opposition, moderator guidance and assertive speaking behaviors, and downgrade or label the evidence completeness of the affected conclusions.

[0038] The completeness of evidence is calculated using the following formula:

[0039]

[0040] In the formula, , , , These are, respectively, the completeness of the transcribed text, the completeness of the quantitative feedback, the completeness of the evidence object, and the completeness of the analysis results; , , , The corresponding weights.

[0041] Beneficial effects

[0042] (1) This invention, by setting up a survey session and corresponding status data, uniformly stores the survey topic, participating users, survey questions, survey units, multimodal stimulus materials, on-site recordings, event markers, ratings and votes, transcribed text, analysis results, and report products in the same session structure. Compared with the existing method of storing images, recordings, tables, and reports separately in surveys, this invention can improve the centralized management capability of survey data and reduce material omissions and context loss.

[0043] (2) This invention divides a survey into multiple survey units, and saves candidate stimulus materials, default selected materials, questions, user feedback, recording tracks, transcribed text, and analysis results under each survey unit, so that each set of user feedback can correspond to a specific survey subject. As a result, the system can avoid mixing user feedback from different scenarios, different questions, or different solutions, and improve the accuracy of subsequent analysis results.

[0044] (3) This invention automatically groups multimodal stimulus materials according to rules such as file order, shooting or creation time, main material anchoring, and time window, and retains the recording, marking, voting, scoring, transcription and analysis results in the original survey unit as much as possible when regrouping. As a result, in cases of supplementing survey materials, splitting equipment or schemes, and removing duplicate materials, the workload of manually reorganizing data can be reduced, and the risk of the collected feedback being destroyed can be reduced.

[0045] (4) This invention records event markers such as start recording, stop recording, question switching, user switching, scoring, voting, start of external editing, and saving of external versions during the on-site survey process, and associates these event markers with the relative time offset of the recording track. This allows continuous recordings to be automatically segmented and assigned to the corresponding survey unit, question, and user after the survey is completed. Compared with the method of manually listening back to the recordings and manually annotating them, this invention can improve the efficiency and accuracy of audio transcription assignment.

[0046] (5) This invention performs noise reduction, slicing, speech recognition, and optional speaker separation on the recordings, and saves the generated transcribed text in a "research unit-question-user" structure, so that the user's original voice can be correlated with specific research questions and specific stimulus materials. As a result, the user's opinions in the report can be traced back to the original audio segments and transcribed text, reducing the bias of conclusions formed solely based on human memory or human summarization.

[0047] (6) This invention preserves the user's quantitative feedback through a rating matrix, voting records, rating history, and withdrawal records, and binds it together with the user's original voice, the current question, and the current stimulus material. Thus, the system can not only record what the user "said," but also what the user "chosen, how many points they gave, and when they made that feedback," thereby improving the evidence completeness of user preference judgment.

[0048] (7) This invention enables multimodal stimulus materials to be converted into displayable, searchable, and analyzable data objects by reading basic metadata, generating caches, extracting objective features, or summarizing images, videos, texts, audio, or other stimulus materials. Taking image-based stimulus materials as an example, the system can read the shooting time, equipment information, size, and exposure information and generate thumbnails. It can also extract features such as brightness, color, saturation, dynamic range, and edge intensity, thereby providing objective evidence for subsequent multi-scheme comparisons and report citations.

[0049] (8) Through similar material identification and objective visual feature detection mechanisms, the system further identifies similar stimulus materials based on the shooting time interval and / or perceptual hash distance, automatically forming a control group. When multiple highly similar materials are collected in a short period of time, the system prompts researchers to select or archive them. This reduces the interference of repeated stimulus materials on user judgment and lowers the possibility of repeated citation of similar materials in subsequent reports.

[0050] (9) This invention responds to user suggestions for modification of the original stimulus material by setting an external processing version, creating a working copy of the original stimulus material, and sending the working copy to an external processing tool; receiving the processed version and processing parameters returned by the external processing tool; re-displaying the processed version in the survey session and collecting participants' ratings, votes, or voice feedback on the processed version; and binding the modification suggestions, processing parameters, processed version, and subsequent feedback into a version verification record. This achieves the binding of the original stimulus material, working copy, processed version, processing parameters, subsequent ratings, user voice, and related event markers. Therefore, when a user suggests modification to a particular solution, the system can record the direction of modification and subsequent feedback, extending the survey conclusions from simple preference descriptions to verifiable version verification results.

[0051] (10) This invention reads the side-by-side files, embedded metadata, or parameter files of the externally processed version and converts the identifiable adjustment items therein into readable parameter records, so that the processed version is no longer just an isolated file, but can be associated with its processing process and user feedback. As a result, researchers can determine whether the specific adjustment direction is effective based on the user's rating of the processed version and the original audio.

[0052] (11) This invention uses a structured corpus construction module to uniformly assemble research units, stimulus materials, users, questions, event markers, transcribed text, ratings and votes, version verification records, material features, and manual notes into a research corpus. This corpus enables the report generation process to prioritize real material paths, user voices, ratings and votes, and version verification records, thereby improving the evidentiary binding force of the research report.

[0053] (12) This invention uses a hierarchical analysis mechanism to segment user comments into judgment segments, extract preferences, extract evaluation dimensions, extract reasons for selection, identify selection relationships, and identify applicable scenarios, enabling fragmented user feedback to be transformed into structured research conclusions. As a result, the system can reduce the cost of manually organizing interview content and improve the consistency of analysis standards between different research units.

[0054] (13) This invention identifies factors such as agreement, soft opposition, moderator guidance, amplification by strong participants, and changes in stance during multi-person discussions, enabling the report to distinguish between independent consensus, recurring viewpoints, individual viewpoints, and viewpoints influenced by interaction. This avoids directly writing individual user viewpoints or viewpoints amplified by on-site interaction as group consensus, thus improving the reliability of research conclusions.

[0055] (14) Through report generation and self-checking mechanisms, this invention can check for issues such as conclusions without evidence, over-inferences, internal contradictions, lack of material citations, and inconsistencies between user expressions and ratings / votes when generating on-site review documents, research and analysis documents, formal reports, or presentations. This improves report delivery quality and reduces the risk of subjective assumptions or broken chains of evidence in reports.

[0056] (15) The processing objects of the present invention are not limited to image materials. Since its core lies in the unified structured processing of survey conversations, survey units, multimodal stimulus materials, on-site events, user feedback and report conclusions, it can be extended to various user survey scenarios such as product prototype testing, interface usability testing, advertising evaluation, content evaluation, consumer experience interviews, teaching review and service process review, etc., and has strong versatility and scalability.

[0057] (16) In actual surveys, researchers may add new questions or follow up, or users may have inconsistent opinions. These dynamic changes may lead to the addition, deletion, or regrouping of stimulus materials. If the system does not have data migration capabilities, these adjustments may result in the loss or mismatch of the original survey data. In this invention, when stimulus materials are added, deleted, or regrouped, the system automatically identifies which materials are shared based on the overlapping relationship of stimulus material identifiers in the old and new survey units, and then migrates all data associated with the material in the original survey unit (event tags, ratings, votes, transcribed text, externally processed versions, analysis results) to the new survey unit.

[0058] (17) In actual research, research reports and raw data may be at risk of leakage during export and sharing, and once leaked, it is difficult to trace the responsible party. This invention implements two security measures when exporting a snapshot of research results: encrypted export and blind watermark tracing. When exporting a snapshot of research results, the system generates a research package based on the research unit, participants, or evidence line range selected by the user, and encrypts and encapsulates the research package; it embeds an invisible watermark into the exported report text or image materials, and the watermark includes at least the export user identifier, export time, or session identifier; the watermark is extracted during subsequent import or verification to determine the source of the research package or report materials. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below.

[0060] Figure 1 This is an overall block diagram of the multimodal survey data processing system according to an embodiment of the present invention;

[0061] Figure 2 This is a flowchart of a multimodal survey data processing method according to an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram illustrating the working principle of the runtime context marking mechanism in an embodiment of the present invention.

[0063] Figure 4 This is a schematic diagram illustrating the determination of audio and video segmentation boundaries according to an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram illustrating the structured evidence aggregation, evidence strength labeling, and evidence gap self-checking of an embodiment of the present invention.

[0065] Figure 6 This is a schematic diagram of external processing tool bridging and product version write-back in an embodiment of the present invention;

[0066] Figure 7This is a schematic diagram illustrating the multi-stage analysis and evidence-bound report generation in an embodiment of the present invention.

[0067] Figure 8 This is a schematic diagram illustrating evidence aggregation and completeness calculation in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with the accompanying drawings and embodiments, further illustrates the invention. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0069] Example 1:

[0070] This embodiment discloses a multimodal data processing method based on user surveys, such as... Figure 2 As shown, it includes the following steps:

[0071] Step S1: Create a user survey session and generate a session identifier for the survey session.

[0072] As the starting point of the multimodal data processing workflow, this step creates a logical container in the system, namely the survey session, to carry a complete user survey task. This logical container has a globally unique session identifier, and all subsequent data related to this survey (audio recordings, videos, texts, ratings, materials, reports, etc.) will be attributed to this identifier.

[0073] After executing step S1, the system generated:

[0074] Session identifier: Usually a UUID or an auto-incrementing numeric string, used to uniquely identify this survey.

[0075] A storage directory was established: a dedicated folder was created for this session, either locally or in the cloud, to store subsequently generated material caches, audio files, audio slices, transcribed text, analysis results, report outputs, and log files.

[0076] Initialize session state: Create a session state object in memory or database. The initial state includes: survey topic, session path, creation time, survey question set (empty), participating user set (empty), survey unit set (empty), current survey unit identifier (empty), current question identifier (empty), current user identifier (empty), event flag set (empty), and report output record (empty).

[0077] Step S2 involves inputting the research plan text (such as a list of questions, task descriptions, and scenario descriptions) prepared in advance by the researchers into the system, and automatically or semi-automatically identifying the research questions and research unit categories from the text, thereby generating the internal question set and research unit structure of the system.

[0078] Researchers can directly input or upload a document containing questions and scenario descriptions into the system. The system scans the document, parses the questions, and identifies sentences containing typical question semantics. For example, sentences containing words like "which one do you like?" or "are you satisfied?" are identified as research questions, while sentences containing words like "scenario" or "task" are identified as research unit categories. All identified questions are stored in the research question set in the session state, with each question assigned a unique question identifier. Finally, based on the parsed unit categories, the system pre-creates several empty research units (containing only unit identifiers and category labels), awaiting the subsequent filling of specific stimulus materials. If the plan text specifies which scenario a question belongs to, the system will associate the question with the corresponding research unit; otherwise, the question defaults to all units or requires manual adjustment later.

[0079] After performing step S2, the survey session will have a set of survey questions (e.g., "Question 1: Which option would you prefer?"), a survey unit skeleton (e.g., "Unit 1: Overall appearance comparison"), and a preliminary mapping relationship between questions and units.

[0080] Step S3: Establish participant and evidence object files. This specifically includes:

[0081] In step S3.1, researchers manually enter participant information (name, ID, role, etc.) or create participants in batches by importing a list. The system assigns a unique participant identifier to each participant. Participant information is stored in the participant user set within the session state.

[0082] Step S3.2: Researchers upload or import the prepared stimulus materials (such as images, videos, prototype files, etc.) into the system via a specified path. The system generates a unique material identifier for each material and then automatically reads the material's metadata, including:

[0083] Image-related information includes: filename, path, format, size, shooting time, device model, exposure parameters, etc.

[0084] Video files include: filename, path, format, duration, resolution, keyframes, etc.

[0085] Text / document types: filename, path, format, title, body summary, paragraph structure, etc.

[0086] The system generates displayable caches, such as thumbnails of images and screenshots of the first frame of videos. For image-based materials, the system can further extract objective features (brightness, color, saturation, edge intensity, etc.) to provide a basis for subsequent identification and grouping of similar materials.

[0087] Step S3.3: Based on the research unit framework created in step S2, the researchers manually or automatically assign the imported stimulus materials to the corresponding research units. For example, the image of Scheme A is assigned to "Unit 1: Appearance Design".

[0088] After completing step S3, the survey session will have complete structured information including the set of participants, the set of evidence objects, material metadata, and the preliminary mapping relationship between materials and survey units.

[0089] Step S4: Acquire audio, and if necessary, simultaneously acquire video or screen material, and record the track start time.

[0090] In step S4.1, the researcher clicks "Start Survey" on the system interface. The system then calls the operating system or hardware's audio / video API to enable microphone, camera, or screen recording. If only audio recording is required, only audio capture will be initiated; if user interface or facial expressions need to be recorded, video or screen capture will be initiated simultaneously.

[0091] In step S4.2, the system records the moment the acquisition started (timestamp T0), which serves as the absolute starting point of the acquisition track. All subsequent event markers' "time offsets" are calculated based on T0.

[0092] In step S4.3, the system automatically writes an event flag of type "Recording Start", which includes the event type, creation time, relative recording offset, and current context index (currently empty or using the default value).

[0093] Step S4.4 involves initialization and continuous maintenance of the context index. The system maintains three variables in memory: the current survey unit, the current question, and the current participant. Before step S4, these three variables are typically set to the first survey unit, the first question, and the first participant, or they can be manually specified by the surveyor.

[0094] After step S4 is completed, audio recording continues, the context index is initialized, and the audio or video file is opened in the background and written to the data stream.

[0095] Step S5: When events such as switching of survey units, questions, participants, or temporary follow-up questions or scoring occur, the system does not interrupt the data collection and directly writes a marker mi carrying the time offset and the current context onto the current track time axis.

[0096] Event tagging is triggered when researchers perform any of the following actions on-site: switching research units, switching questions, switching participants, initiating follow-up questions, users rating, voting or selecting, starting / ending external editing, or other custom events. The system immediately generates an event tag object containing the fields: event type, system time of the event, offset relative to the recording start time, snapshot of the current context index, material identifier (optional, for specific stimulus materials), and score / option (optional, for ratings / votes). The system then appends the generated event tag to an ordered list (in memory or a database) associated with the recording track. If the event involves a context switch (such as a question switch), the system simultaneously updates the current research unit identifier, current question identifier, and current participant identifier in memory so that subsequent event tags use the latest context.

[0097] Step S5 is one of the inventive points of this application, solving the technical problem in traditional surveys where recording time requires interrupting audio recording or manually comparing the timeline afterwards. Figure 3 As shown, without stopping the recording / video acquisition, important nodes in the survey process are recorded by writing lightweight event markers. Each marker carries a time offset and the current context index (survey unit ID, question ID, participant ID), which enables the continuous audio and video streams to be automatically segmented into meaningful segments and accurately assigned to the corresponding survey context.

[0098] In step S6, the system continuously maintains the current context index to ensure that all subsequent feedback (ratings, votes, comments, etc.) can be automatically bound to the correct context.

[0099] The system continuously tracks data in real time throughout the survey collection process. Specifically, with the writing of the event marker in step S5, it synchronously updates the three variables in memory: the current survey unit identifier, the current question identifier, and the current participant identifier. It then passes the latest context index to all submodules that need to bind context, ensuring that data generated by submodules automatically carries the correct context. During the update, the system may briefly lock the context variables to prevent other threads from reading inconsistent values. After the update is complete, the lock is released. Whenever an operation requiring context binding occurs (such as a user clicking the rating button), the system directly reads the current three variables from memory and writes them to storage as part of the data.

[0100] Step S7: Based on the participants' ratings, rankings, or votes for specific samples, save the data using a composite key of the current unified context index and the evidence object identifier to form a feedback matrix. If external modifications are involved (such as prototype adjustments), the system locates the object based on the current context, calls external tools, and writes the returned verification artifacts, version numbers, parameter summaries, etc., back into the current context to form a version time series. This includes:

[0101] Step S7.1: Quantify the received feedback information. Participants rate, vote, or sort the currently displayed stimulus materials on the system interface; the system reads the current survey unit identifier, current question identifier, and current participant identifier from memory, and determines the identifier of the stimulus material that the user is currently evaluating, storing it as a mind combination key of (current survey unit identifier, current question identifier, current participant identifier, evidence object identifier) ​​to form a cross-comparable matrix, and storing the rating value or voting option into structured data, and simultaneously writing an event tag of type "rating" or "voting" (step S5 has been completed).

[0102] Step S7.2 involves making modifications based on external tools. For example... Figure 6 As shown, when a user suggests modifications, the system can call external tools (such as an image editor) to generate a new version and write back the new version file, modified parameters, and the user's secondary rating of the new version to the same context, forming a time sequence of version evolution.

[0103] The system copies the original stimulus material as a working copy, recording the original path and baseline version identifier; it also writes an event marker "External Editing Begins," carrying the current context and material identifier. The system launches an external tool or sends the working copy to it for processing. During processing, the system may be in a waiting state, but data acquisition such as audio recording continues. After the external tool completes its modifications, the system imports the final version file and attempts to read side-by-side files, embedded metadata, or parameter files to extract identifiable adjustment parameters. A version record is then generated, containing the version identifier, the final version file path, a parameter summary, and the processing time. Users rate or comment on the modified version again, and the system stores this secondary feedback using the same composite key. Finally, an event marker "External Editing Saves" is written. Within the same context, the feedback data for the original version and multiple modified versions are arranged chronologically, forming a version evolution trajectory.

[0104] After completing step S7, the system accumulates quantitative feedback data such as ratings and votes indexed by composite keys, supporting cross-queries by multiple dimensions such as questions, participants, and materials. At the same time, for materials that have been modified externally, the system records the original version, modified parameters, final version, and user feedback on each version, forming a complete version verification record. In addition, all rating, voting, and external editing events are accompanied by corresponding timestamps and context markers, which facilitates replay and alignment afterward.

[0105] Step S8: After acquisition, the system uses the context marker as the anchor point ma to search for boundary markers mb that meet the conditions (such as stop markers or markers where context changes have occurred). The audio and video segments between ta and tb are extracted and submitted for transcription, such as... Figure 4 As shown.

[0106] After all survey data collection is completed (audio / video recording has stopped), the system no longer deals with continuous long audio clips but needs to cut them into multiple meaningful short segments. The cutting is based on the event markers written in steps S5 and S6. Each event marker records the current time offset and context. The system uses a given event marker as an anchor point *ma* and then searches for the next boundary marker *mb*. Boundary markers are typically stop markers (e.g., recording stops) or markers indicating a change in context (e.g., question switching, participant switching). The time interval between two markers, *ta* to *tb*, constitutes an independent audio / video segment, and all content within this segment belongs to the same survey context. Specifically,

[0107] Step S8.1: The system reads all event markers accumulated during the acquisition process and sorts them in ascending order by time offset.

[0108] Step S8.2: Using the first event marker (recording start marker) as the first starting anchor point ma, traverse the marker list backwards from ma, and find a marker that satisfies any of the following conditions as the boundary marker mb:

[0109] The marker type is recording stop, question switch, participant switch, survey unit switch, or temporary follow-up question (optional, depending on the configuration).

[0110] Step S8.3: Read the time offset of ma as the start time ta, and read the time offset of mb as the end time tb. Then, use an audio / video processing tool to extract the portion from ta to tb from the original recording file to generate a separate audio file.

[0111] Step S8.4: Bind the extracted fragment to the context index (current survey unit identifier, current question identifier, current participant identifier) ​​recorded in ma or mb, usually to the context of ma.

[0112] Step S8.5: The extracted audio segments are sent to the speech recognition engine for transcription (this step completes the segmentation and is ready to submit, providing data conditions for the transcription in S9-S10 to form readable text and classify and store it).

[0113] Step S8.6: Using mb as the new starting point anchor, repeat steps S802-S805 until a recording stop marker is encountered.

[0114] The process of step S8 is shown in a slice: starting point ma = participant switching (30s), find the boundary → question switching (60s). The 30s~60s slice belongs to the context (unit1, q1, U002), that is (research unit, question, participant).

[0115] After completing step S8, the system obtains several independent audio and video clip files, each of which is clearly assigned to a specific research context (research unit, question, participant).

[0116] Step S9 involves submitting the segmented audio and video fragments from Step S8 to the speech recognition engine for transcription into readable text. The generated transcribed text is then automatically bound and categorized with its corresponding context index. This ensures that each segment of user speech becomes structured textual evidence with clearly defined attribution information. Specifically,

[0117] From the output of step S8, receive the segmented audio file and its bound context index one by one. Send the audio segments to the speech recognition engine to perform noise reduction (optional), speech recognition, and speaker separation (optional) operations. The speech recognition engine returns the transcribed text. The system associates the transcribed text with the context index bound to the segment in step S8, that is, writes the transcribed text into the session data structure, storing it as a transcription record with (current survey unit identifier, current question identifier, current participant identifier) ​​as the key.

[0118] In addition to transcribing the text, the system can also save the confidence score of speech recognition, the start and end timestamps of each word, and speaker tags (if separation was performed) for subsequent detailed analysis. The above steps of submitting transcription, obtaining transcription results, binding context indexes, and saving auxiliary information are repeated for all segments until all segments are transcribed.

[0119] After executing step S9, the system accumulates a set of transcribed texts bound to the context index, such as (unit1, q1, U001) → "I think solution B is clearer, but solution A is more natural."

[0120] Step S10: Each segment of transcribed text output from step S9 is categorized, merged, and persistently stored according to its corresponding unified context index (research unit identifier, question identifier, participant identifier), forming a structured transcribed dataset in the format: Research Unit-Question-Participant. This includes:

[0121] Step S10.1: Receive each transcription record one by one from step S9. Each record contains: transcribed text, context index, audio segment path, recognition confidence, and timestamp.

[0122] In step S10.2, in the transcription dataset of the session, check if a record with the key (unit1, q1, U001) already exists. If it does not exist, create a new record, use the transcribed text as the initial content, and store metadata such as audio path and confidence score. If it exists, append the current transcribed text to the existing text (separated by newline character or space), and update the audio path list (keeping multiple segment paths) and the average confidence score.

[0123] Step S10.3: For the merged record, save the following information: complete transcribed text, a list of all associated audio segment paths, average confidence or minimum confidence, total speaking time (sum of all segment durations), and speaking time range (earliest start time to latest end time), and write the updated transcribed dataset back to the session state storage.

[0124] Step S10.4: Repeat steps S1002-S1003 for all transcribed records until all fragments have been processed.

[0125] After completing step S10, a structured transcription dataset organized by context index is formed in the system. Each participant's complete speech under each question is integrated into a single record, and a reference to the original audio is retained for subsequent manual review.

[0126] In step S11, the system performs aggregation operations on a unified context index basis.

[0127] The system uses the (research unit identifier, question identifier, participant identifier) ​​triple as the basic unit of aggregation. That is, each "research unit-question-participant" combination generates a structured evidence row, such as... Figure 5 As shown.

[0128] The system collects the following information: survey unit information (including basic information such as the name, category, and description of the current unit), question information (reading the text content of the current question), participant information (reading the name, number, and other basic information of the current participants), transcription text (extracting the complete transcription text and audio clip path list corresponding to the context key from the transcription dataset generated in step S10), stimulus material information (finding all stimulus materials related to the current question under the current unit from the survey unit organization, and reading their identifiers, paths, types, metadata, etc.), participant feedback (extracting quantitative data such as ratings, votes, and rankings corresponding to the context key and material identifier from the feedback matrix), external product versions (extracting all version information corresponding to the context key and material identifier from the version verification record), and event markers (filtering all markers within the current context key range from the event marker list, such as rating markers and switching markers, for subsequent timeline reconstruction). All of the above information is packaged into a JSON object or similar structured data and stored in the structured evidence row set of the session for direct access by subsequent analysis modules.

[0129] After step S11 is completed, the system generates a structured evidence row table. Each row in the table is an independent evidence package containing all multimodal evidence in that context.

[0130] Step S12: Based on structured evidence, the system performs multi-stage semantic analysis (including preference extraction and trade-off analysis). Particularly for multi-participant scenarios such as focus groups, the system conducts influence auditing, identifying agreement, soft opposition, moderator guidance, and speeches. Based on formulas... Completeness of evidence in computational context .like Figure 7 As shown, specifically,

[0131] From the output of step S11, read each evidence line one by one and perform a three-level analysis:

[0132] The first layer of analysis involves decision segmentation. NLP techniques (such as dependency parsing and semantic role labeling) are used to split the transcribed text into independent decision segments.

[0133] For example: The input "Option B is clearer, but Option A is more natural; I might choose B when sending it to a friend, but I might choose A for long-term use" can be broken down into four parts:

[0134] Segment 1: Option B is clearer (Dimension: Clarity, Preference: B).

[0135] Segment 2: Option A is more natural (Dimension: Naturalness, Preference: A).

[0136] Segment 3: Social posting scenario → Choose B (Scenario: Social, Preference: B).

[0137] Segment 4: Long-term use scenario → Choose A (Scenario: Long-term, Preference: A).

[0138] The second layer of analysis involves extracting information based on dimensions, preferences, reasons, and trade-offs. Structured information is extracted from each judgment fragment: evaluation dimensions (clarity, naturalness, aesthetics, etc.), preferred object (option A / B / C), reasons for selection, applicable scenarios, and trade-off relationships. Conditional preferences are labeled, such as "social scenarios → B, long-term scenarios → A".

[0139] The third layer of analysis involves auditing social impact and overattribution. In multi-person discussion scenarios, the analysis examines the order and relevance of the speakers' content on the same issue, including checks for agreement, soft objection, moderator guidance, and dominant participant influence. If participant B's statement is highly similar to participant A's and follows closely behind, it is marked as "potentially agreeable"; if participant B first affirms and then reverses course, it is marked as "soft objection"; if the moderator's question contains biased language, it is marked as "moderator guidance." The analysis also tracks the speaking time, number of times, and frequency of interruptions for each participant to identify the dominant participant and mark their viewpoint as "potentially amplified." Conclusions affected by these influences are downgraded (e.g., "Confidence reduced due to interaction").

[0140] Finally, the completeness of the evidence is calculated. For each context c, four metrics are calculated:

[0141] Based on factors such as the number of words transcribed, whether it contains complete sentences, and the presence of obvious noise;

[0142] Based on factors such as the number of words transcribed, whether it contains complete sentences, and the presence of obvious noise;

[0143] Based on factors such as the number of associated stimulus materials, the richness of metadata, and the availability of external versions;

[0144] Based on whether structured information such as preferences, dimensions, and trade-offs were successfully extracted.

[0145] Substitute into the formula Calculate the total score and... This serves as a label for the strength of evidence in that context (e.g., high, medium, low).

[0146] Write the results of each layer of analysis (list of judgment fragments, preference structure, social impact markers, and evidence completeness) back into the evidence row for use in step S13 report generation.

[0147] After performing step S12, each line of structured evidence is supplemented with a list of judgment fragments (each fragment includes text, evaluation dimension, preference object, reason, scenario, and confidence level), a trade-off relationship (e.g., social scenario preference B, long-term scenario preference A), social impact markers (e.g., agreement, soft opposition, moderator guidance, assertive speech), and an evidence completeness score. This transforms the original transcribed text and scoring data into structured conclusions that can be directly used to generate reports, with each conclusion accompanied by labels indicating the strength and credibility of the evidence.

[0148] Step S13: When generating the report, if the contextual evidence supporting a conclusion is insufficient or influenced by strong opinions, the system will automatically downgrade the conclusion or mark it as "needs further verification" to prevent isolated cases from being mistakenly written as consensus. The report generation module can also perform self-checks to identify issues such as conclusions lacking evidence, over-inferences, internal contradictions, lack of stimulating material citations, and inconsistencies between user verbal expressions and ratings / votes. Figure 7 As shown, specifically,

[0149] In step S13.1, the system determines the chapter structure based on a preset report template (such as a site review document, a formal research report, or a presentation); then, it iterates through each conclusion from the analysis results output in step S12 and checks the evidence completeness score corresponding to that conclusion. Set pre- and post-thresholds. , ( < ),when > When deemed to have high strength of evidence, the conclusion can be included in the main body of the report as a consensus, citing specific evidence (transcribed text, scoring data, and material source). When the evidence is deemed of medium strength, the conclusion is written into the report but marked as "majority opinion" or "repeated," and the source of the evidence is attached; when... If the evidence is deemed to be of low strength, the conclusion is not included in the main body of the report, but is placed in the "To be further verified" appendix, with the reasons noted (such as the statement being too short, lack of scoring, or only one person's opinion).

[0150] Step S13.2: Check whether the conclusion is marked as being influenced by social factors (such as agreement or moderator guidance). If so, mark "Influenced by interaction" next to the conclusion and explain the specific source of influence; if the conclusion comes from an individual case (mentioned by only one participant), mark it as "Individual case viewpoint" and do not generalize it as a group consensus.

[0151] Step S13.3: For each adopted conclusion, the system uses natural language generation technology to convert it into a fluent declarative sentence. For example:

[0152] Input: {Conclusion: "Option B is clearer", Evidence: "Zhang San: 'Option B is clearer' (Score: 9)", Strength of evidence: "High"}

[0153] Output: "In terms of clarity, participants generally considered Option B to be superior. For example, participant Zhang San stated that 'Option B is clearer' and gave it a high score of 9."

[0154] Regarding conditional preferences (trade-offs), the system will specifically explain the scenario-dependent nature. For example: "In a social sharing scenario, participants tend to choose option B; while in a long-term use scenario, they prefer option A."

[0155] Step S13.4: After the report is generated, the system performs checks for conclusions lacking evidence, over-inference, internal contradictions, missing citations, and inconsistencies. Specifically, the lack-evidence check scans the automatically generated report text to find conclusions that lack any underlying evidence (such as transcripts, rating data, or stimulus materials). Once found, such conclusions are immediately marked as "lacking supporting evidence." The over-inference check verifies the amount of supporting evidence for conclusions claiming to represent "group consensus." If the evidence comes from only 1 to 2 participants, it is judged as "potentially over-inference." The internal contradiction check compares the analytical conclusions of different sections of the report or different participants to detect logical conflicts. The missing citation check verifies whether the report text has omitted citations of key stimulus materials. The inconsistency check deeply compares participants' verbal expressions with objective rating data. If a participant verbally praises a solution but gives it a low score, it is marked as "user expression and rating inconsistent, requiring manual review."

[0156] After completing step S13, the system outputs one or more generated report files, along with an evidence index appendix listing the evidence line ID, original text, scoring data, and material path for each conclusion, facilitating reader verification. For example... Figure 8 The diagram shown illustrates evidence aggregation and completeness calculation. The survey input corresponds to steps S8-11, data preprocessing and aggregation; context binding and evidence row aggregation correspond to step S11, constructing evidence rows and structured evidence rows; completeness calculation corresponds to step S12, auditing the social impact of evidence completeness; and constraint output corresponds to step S14, report generation and self-inspection.

[0157] Step S14: Results Upload and Asynchronous Collaboration. During the research preparation phase (Step S1: Creating a research session; Step S2: Importing the research outline and analyzing research units; Step S3: Establishing participant and evidence object files), standardized research templates are published in the cloud and researchers can pull them to generate local research sessions. During the on-site execution phase (Step S4: Starting data collection; Step S5: Writing event markers; Step S6: Continuously maintaining the context index; Step S7: Feedback collection and output writing), research data, images, audio recordings, scores, and scene records are all stored and processed offline locally to avoid network instability affecting the research process. During the debriefing phase (Step S8: Automated segmentation; Step S9: Speech transcription; Step S14: ...), ... (Steps S10: Transcription Result Classification; S11: Aggregation of Structured Evidence Lines; S12: Multi-Stage Analysis; S13: Report Generation and Self-Check) Researchers upload the selected result snapshots, structured scores, transcribed text, key images, and analysis conclusions to the cloud. Team members can then view the snapshots in read-only mode, provide asynchronous annotations, participate in thread discussions, receive @ reminders, and mark processing status. Annotated data is stored separately from the original survey data to avoid contaminating the original evidence. During the sharing phase, the system generates read-only sharing links with access control, validity period control, and access scope control, enabling secure sharing of individual projects or scenarios. Through these methods, without introducing real-time multi-user editing, complex conflict merging, or continuous network dependencies, the system achieves minimal closed-loop collaboration in enterprise user surveys, enabling template reuse, offline data collection, result review, team review, and secure sharing.

[0158] Example 2:

[0159] This embodiment discloses a multimodal data processing system based on user surveys. The system is deployed on a survey terminal device equipped with a microphone, camera, display screen, and network communication module. The system comprises 12 core modules: a survey session management module, a survey plan parsing module, a multimodal material access module, a survey unit organization module, a field data collection module, an event marking module, a multimodal preprocessing module (including audio / video processing, visual / text feature processing, and scoring / voting modules), a structured evidence line construction module, a multi-stage analysis module, a report generation module, a versioned storage module, and a collaboration management module. Figure 1 As shown.

[0160] The survey session management module is responsible for creating and maintaining the global state of a survey task. When a survey session is created, this module generates a unique session identifier and establishes a corresponding storage directory, either locally or in the cloud, to store all subsequently generated data files. The module initializes the session state object, including the survey topic, session path, creation time, set of survey questions, set of participating users, set of survey units, current survey unit identifier, current question identifier, current participant identifier, set of event markers, and report output records. During survey execution, this module continuously maintains the above three current identifier variables and updates them synchronously when the event marker module writes context change markers. It also provides a query interface for other modules to use.

[0161] The survey plan parsing module receives survey plan text, question lists, or task descriptions input by survey personnel, and automatically identifies and extracts survey questions and survey unit categories. This module matches typical question formats using a keyword rule base; for example, sentences containing words like "which one do you like?", "Are you satisfied?", or "Why?" are identified as survey questions; sentences containing words like "scenario," "task," or "interface operation" are identified as survey unit categories. For ambiguous content that cannot be automatically identified, the module uses a semantic parsing model for auxiliary judgment, or provides a manual confirmation interface for survey personnel to manually annotate. The identification results are written into the session state maintained by the survey session management module.

[0162] The multimodal material access module is used to import stimulus materials for user evaluation or experience. This module generates a unique material identifier for each material and calls the corresponding metadata reader based on the material type: for image materials, it reads the filename, path, format, size, shooting time, device model, and exposure parameters; for video materials, it reads the filename, path, format, duration, resolution, and extracts keyframes; for text or document materials, it reads the title, body summary, and paragraph structure. The module also generates displayable caches, such as thumbnails for images and screenshots of the first frame of videos. For image materials, the module further extracts objective features such as brightness, color, saturation, dynamic range, and edge intensity. In addition, the module identifies burst or similar groups of images based on shooting time proximity and perceptual hash distance, prompting researchers to select or archive them.

[0163] The research unit organization module is used to organize imported stimulus materials into multiple research units according to preset rules. This module supports sequential grouping, time window grouping, master material anchoring grouping, manually planned grouping, similarity grouping, or combinations thereof. Taking sequential grouping as an example, if there are schemes A, B, and C, each containing multiple materials, then the first material of each scheme will form research unit 1, the second material will form research unit 2, and so on. Each research unit includes a unit identifier, unit name, candidate material list, default displayed material, and associated question list. When materials are added, deleted, archived, or regrouped, the module migrates the recording tracks, event markers, scores, votes, transcribed text, analysis results, notes, and version verification records from the original research unit to the most matching new research unit based on the overlap of material identifiers in the old and new research units, thus maintaining the continuity of evidence.

[0164] The on-site data acquisition module initiates and manages the real-time acquisition of multimodal data during the survey execution process. Upon receiving the "Start Acquisition" command from the survey personnel, this module activates microphone recording, and if necessary, simultaneously initiates camera video recording or screen recording, recording the start time of the acquisition track as the zero point for the time offset of all subsequent event markers. During acquisition, the module continuously writes audio or video data streams to temporary files or memory buffers, and maintains the acquisition status for running, paused, and stopped data. Upon receiving the "Stop Acquisition" command, the module stops all acquisition devices, closes file streams, saves the original audio and video files, and transmits the relevant information of the acquisition track to the event marker module and the multimodal preprocessing module.

[0165] The event tagging module responds to various runtime events during the survey execution process, writing event tags onto the collection timeline without interrupting data collection. The types of events this module monitors include recording start, recording stop, question switching, participant switching, survey unit switching, scoring, voting, score withdrawal, material selection, external editing start, and external editing save. When an event is triggered, the module immediately generates an event tag object, containing the event type, the system time of the event occurrence, the millisecond offset relative to the collection start time, a snapshot of the current context index, and optional evidence object identifiers and scores or options. The module appends the event tag to an ordered tag list and notifies the survey session management module to update the current context index when the event involves context changes. After collection is completed, the complete event tag list is persistently stored.

[0166] The multimodal preprocessing module performs analyzable processing on the acquired raw multimodal data. This module first obtains an ordered list of event markers from the event marker module. Starting with the recording start marker, it sequentially traverses the event markers, using adjacent context change markers or recording stop markers as boundaries to determine the start and end times of each audio segment. It then extracts each segment from the original recording file, generating independent audio files. After performing noise reduction on each audio segment, the module sends it to the speech recognition engine for transcription, obtaining the transcribed text and binding it to the context index corresponding to the segment. If the same context index corresponds to multiple non-contiguous audio segments, the module concatenates the transcribed text of multiple segments into a complete speech record, retaining a path list of all audio segments. For video materials, the module extracts keyframes, duration, and segment identifiers; for image materials, it generates thumbnails and extracts objective features; for text materials, it performs text extraction, summarization, and keyword recognition.

[0167] The structured evidence row construction module aggregates multi-source data from the survey session according to a unified context index, assembling them into complete structured evidence rows. This module obtains survey unit information, question information, and participant information from the survey session management module; transcribed text and audio clip paths organized by context index from the multimodal preprocessing module; scoring matrices and voting records from the on-site data collection module or scoring and voting module; version verification records from the external processing version module; and an event tag list from the event tagging module. The module uses runtime context indexes as units to populate the corresponding fields of the evidence row, including survey unit name, question text, participant name, transcribed text, audio clip path, stimulus material identifier and metadata, score value or voting option, version identifier and parameter summary, and event tag list, generating a unique evidence row identifier for each row. All evidence rows are stored in the session's structured evidence row set, which can be directly read by the hierarchical analysis module and report generation module.

[0168] The multi-stage analysis module performs multi-level semantic analysis based on structured evidence lines. The first layer of analysis segments the transcribed text of each evidence line into judgment segments, using natural language processing technology to break down the entire speech into independent judgment units. Each unit contains a clearly defined evaluation object, evaluation dimension, and preference tendency. The second layer of analysis extracts structured information from each judgment segment, including evaluation dimension, preference object, reasons for selection, applicable scenarios, and trade-off relationships. For speech records containing conditional preferences, conditional preferences are considered rather than a single winner. The third layer of analysis performs a social impact audit, analyzing the speaking sequence and semantic relevance of multiple participants on the same issue, identifying interaction factors such as agreement, soft opposition, moderator guidance, and dominant participants. The fourth layer of analysis calculates the evidence completeness of each evidence line. The score is obtained by weighted summation of four indicators: completeness of transcribed text, completeness of quantitative feedback, completeness of evidence object, and completeness of analysis results. The analysis results are written back to the corresponding structured evidence line.

[0169] The report generation module generates evidence-bound research reports based on structured evidence lines and stratified analysis results. This module determines the chapter structure according to a preset report template, iterates through each analytical conclusion, and performs evidence constraint checks: conclusions with high evidence strength are written into the main body of the report; conclusions with medium evidence strength are marked as "majority view" or "repeated"; conclusions with low evidence strength are moved to the "to be further verified" appendix; conclusions marked as having social impact are accompanied by annotations of the sources of impact; individual viewpoints are marked as "individual viewpoints." The module uses natural language generation technology to convert structured conclusions into declarative sentences and embeds evidence citations within these sentences. After report generation, the module performs self-checks, including checks for conclusions without evidence, over-inference, internal contradictions, missing material citations, and inconsistencies between statements and actions, and appends the self-check results to the report draft as annotations or revision marks. The final output report includes an evidence index appendix, listing the evidence line identifier, original text, scoring data, and material path for each conclusion.

[0170] The versioned storage module saves the complete state, intermediate deliverables, and final output of a survey session. At each stage of the survey, the module automatically saves the session state, transcription results, analysis results, and report deliverables as independent versions, recording the version number, save time, and version description each time. The module allows researchers to manually trigger version saving to create backups at critical points. When the survey structure needs to be reused for a new round of surveys, the module copies structured information such as the survey topic, questions, users, survey units, and stimulus materials, while clearing or resetting runtime data such as recordings, tags, ratings, transcriptions, analyses, and reports. The module also supports loading and comparing historical versions, facilitating researchers to trace changes in the data processing process.

[0171] The collaboration management module is used to publish standardized research templates from the cloud during the research preparation phase, allowing researchers to pull them to generate local research sessions. During the on-site execution phase, it ensures that research data, images, audio recordings, scores, and scene records are all stored and processed offline locally, preventing network instability from affecting the research process. During the debriefing phase, it receives filtered snapshots of results, structured scores, transcribed text, key images, and analysis conclusions uploaded by researchers, providing team members with read-only viewing, asynchronous annotation, thread discussion, @mentions, and processing status marking functions based on these snapshots. Annotated data is stored separately from the original research data to avoid contaminating the original evidence. During the sharing phase, it generates read-only sharing links with access control, validity period control, and access scope control, enabling secure sharing of individual projects or scenarios. Through these modules, a minimal closed-loop collaboration mechanism is achieved in enterprise user research, enabling template reuse, offline data collection, result debriefing, team review, and secure sharing without introducing real-time multi-user editing, complex conflict merging, or continuous network dependencies.

[0172] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal data processing method based on user surveys, characterized in that, Includes the following steps: S1, create a user survey session and maintain a runtime context index including the current survey unit identifier, the current question identifier, and the current participant identifier; S2, during the survey execution process, in response to runtime events, an event marker is written on the acquisition timeline without interrupting multimodal data acquisition. The event marker carries a time offset and the runtime context index. S3. After the data collection is completed, the collected multimodal data is segmented according to the event markers, and the segmented data fragments are bound to the corresponding runtime context indexes to generate transcribed text. S4, using the runtime context index as a unit, aggregate the transcribed text, corresponding stimulus materials, user feedback data, and external verification products to generate a structured evidence row; S5. Based on the structured evidence row, perform multi-stage analysis, which includes social impact audit and evidence completeness calculation, and generate analysis conclusions with evidence completeness markers. S6. Based on the correlation between the analysis conclusions and the structured evidence rows, perform report generation and self-checking of evidence constraints, and output a research report containing evidence tracing.

2. The multimodal data processing method according to claim 1, characterized in that, In step S2, the runtime events include at least one of the following: survey unit switching, question switching, participant switching, rating, voting, external editing start, and external editing save.

3. The multimodal data processing method according to claim 1, characterized in that, In step S3, the binding includes: The start and end times of an audio / video segment are determined based on the time offset between two adjacent event markers. The audio / video clips are subjected to speech recognition to generate transcribed text; Associate the transcribed text with the runtime context index in the event tag corresponding to the start time.

4. The multimodal data processing method according to claim 1, characterized in that, In step S4, the structured evidence row includes at least: survey unit information, question information, participant identifiers, transcribed text, stimulus material identifiers and metadata, rating and voting matrix, external product version records, and event tag list.

5. The multimodal data processing method according to claim 1, characterized in that, In step S5, the social impact audit includes: analyzing the speaking time sequence and semantic relevance of different participants on the same issue, identifying echoing, soft opposition, moderator guidance, and assertive speaking behaviors; and downgrading or labeling the evidence completeness of conclusions affected by the above behaviors.

6. The multimodal data processing method according to claim 1 or 5, characterized in that, The completeness of the evidence is calculated using the following formula: ， In the formula, , , , These are, respectively, the completeness of the transcribed text, the completeness of the quantitative feedback, the completeness of the evidence object, and the completeness of the analysis results; , , , The corresponding weights.

7. The multimodal data processing method according to claim 1, characterized in that, In step S6, the report generation and self-checking of the evidence constraints include: classifying the analysis conclusions into consensus, majority opinion, individual opinion, and opinion to be verified based on the evidence completeness markers and social impact audit results; scanning the generated report text to detect conclusions lacking evidence support, internal contradictions, and inconsistencies between user verbal expressions and ratings, and outputting the self-checking results.

8. A multimodal data processing system based on user surveys, characterized in that, include: The survey session management module is used to create and maintain user survey sessions, as well as a runtime context index consisting of the current survey unit identifier, the current question identifier, and the current participant identifier. The event tagging module is used to write event tags carrying time offsets and the runtime context index on the acquisition timeline in response to runtime events during the survey execution process, without interrupting multimodal data acquisition. The multimodal preprocessing module is used to segment the multimodal data according to the event markers after the acquisition is completed, and to transcribe the segmented data fragments and bind the transcribed text to the corresponding runtime context index. The structured corpus construction module is used to aggregate transcribed text, stimulus materials, user feedback data, and external validation products into structured evidence rows, using the runtime context index as the unit. The hierarchical analysis module is used to perform multi-stage analysis based on the structured evidence line, the multi-stage analysis including social impact audit and evidence completeness calculation, and generate analysis conclusions with evidence completeness markers; The report generation module is used to perform report generation and self-checking based on the correlation between the analysis conclusions and the structured evidence rows, and output a research report containing evidence tracing.

9. The system according to claim 8, characterized in that, The event tagging module supports runtime events including at least one of the following: survey unit switching, question switching, participant switching, rating, voting, external editing start, and external editing save.

10. The system according to claim 8, characterized in that, The hierarchical analysis module is also used to: analyze the speaking time sequence and semantic relevance of different participants on the same issue, identify echoing, soft opposition, moderator guidance and assertive speaking behaviors, and downgrade or label the evidence completeness of the affected conclusions. The completeness of evidence is calculated using the following formula: ， In the formula, , , , These are, respectively, the completeness of the transcribed text, the completeness of the quantitative feedback, the completeness of the evidence object, and the completeness of the analysis results; , , , The corresponding weights.

Images