Multi-scene target detection and behavior recognition method and system based on deep learning

Abstract

The invention provides a multi-scene target detection and behavior recognition method and system based on deep learning, and relates to the technical field of software development behavior analysis, and the method comprises the steps: obtaining a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment, and forming a multi-modal development behavior data flow; performing development scene identification on the multi-modal development behavior data flow to obtain a scene state identifier; constructing a semantic perception feature extractor, performing syntax structure analysis and semantic dependency modeling on the code editing operation sequence, extracting an abstract syntax tree structure and a program execution path diagram, and combining to obtain code structured features; performing code mode matching and abnormal mode positioning to obtain a target code segment detection result; extracting operation time sequence features and carrying out cross-modal feature alignment on the operation time sequence features and the code structured features to obtain a unified behavior vector; and performing time sequence dependence reasoning and behavior pattern classification to obtain a behavior recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software development behavior analysis, and particularly relates to a multi-scene target detection and behavior recognition method and system based on deep learning. BACKGROUND

[0002] With the continuous improvement of software development complexity, intelligent analysis and understanding of the programmer's programming behavior have become an important means to improve software quality and development efficiency. Traditional software development process monitoring mainly relies on static code analysis tools and simple log recording systems. These methods can capture basic data such as code submission records, compilation error information, etc., but it is difficult to deeply understand the real intention and behavior pattern of the developer. In recent years, with the rapid development of deep learning technology, researchers have begun to try to apply neural network models to code understanding, defect detection and development behavior analysis, etc. By learning the pattern features in a large-scale code corpus, automatic understanding of code semantics and intelligent recognition of development behavior are realized.

[0003] Existing code analysis methods mainly process based on a single modality of data source, usually only focusing on the static features of the code text itself, while ignoring the dynamic information such as interface interaction behavior and file operation behavior generated by the developer during coding. This single-modal analysis method cannot fully capture the complete semantics of the development behavior, resulting in limitations in understanding the real intention of the developer, and it is difficult to accurately distinguish between programming operations with similar surfaces but different actual purposes, affecting the accuracy of behavior recognition and the comprehensiveness of code problem detection. The existing target detection method lacks the ability to perceive the context of the development scene when processing code data, and usually uses a unified feature extraction strategy to process all types of code fragments, without considering the differences in code features in different development stages and task scenarios. In the actual development process, the same code pattern has completely different semantic meanings in different scenarios, for example, temporary code in the test stage and formal code in the production environment have different quality requirements, and the existing method cannot dynamically adjust the detection strategy according to the scene, which is prone to false detection and missed detection problems. SUMMARY

[0004] The embodiment of the present application provides a multi-scene target detection and behavior recognition method and system based on deep learning, which can solve the problems in the prior art.

[0005] The first aspect of the embodiment of the present application provides a multi-scene target detection and behavior recognition method based on deep learning, comprising: acquiring a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and forming a multi-modal development behavior data stream; scene state identification is obtained by analyzing development stage semantic features and task context features; Based on the scene state identification, a semantic perception feature extractor is constructed, the code editing operation sequence is parsed by the semantic perception feature extractor for syntax structure and semantic dependency modeling, the abstract syntax tree structure and program execution path graph of the code segment are extracted, and the code structured features are obtained by combination; the code structured features are matched with code patterns and abnormal patterns are located to identify the target functional unit position and potential defect position in the code, and a target code segment detection result is obtained; Based on the target code segment detection result, operation timing features associated with the target code segment are extracted from the interface interaction event sequence and the file access record sequence, the operation timing features and the code structured features are cross-modal feature aligned to obtain a unified behavior vector; the unified behavior vector is subjected to timing dependency reasoning and behavior pattern classification to identify the coding behavior type of the developer, and a behavior recognition result is obtained.

[0006] Obtaining code editing operation sequence, interface interaction event sequence and file access record sequence in software development environment and forming multi-modal development behavior data stream includes: In the software development environment, the code editor interface is monitored in real time, the code editing operation events triggered by the developer are captured, the operation type identification and operation position information in the code editing operation events are analyzed, and the code editing operation sequence is arranged in time sequence; Intercept the event of the graphical interface layer, get the interaction trigger record of the interface control, extract the control identification and interaction action type in the interaction trigger record, and arrange the interface interaction event sequence in time sequence; Call monitoring is performed on the file system access interface, record the read operation and write operation of the file, extract the file path identification and access timestamp corresponding to the read operation and write operation, and arrange the file access record sequence in time sequence; The code editing operation sequence, the interface interaction event sequence and the file access record sequence are time-aligned according to a unified time reference, a cross-modal time index structure is constructed, and the operation records in the same time window in the three sequences are associated and bound based on the cross-modal time index structure to form the multi-modal development behavior data stream.

[0007] The scene state identification is obtained by analyzing development stage semantic features and task context features; extract operation type distribution information and operation frequency change information from the multi-modal development behavior data stream, count the occurrence times of each operation type in the operation type distribution information, calculate the change rate of the operation frequency change information in the time dimension, determine the operation type with the highest occurrence times as the dominant operation type, and combine the dominant operation type and the change rate to obtain the development stage semantic feature; According to the control calling relationship and file access path dependency relationship in the multi-modal development behavior data stream, a task association graph is constructed, the nodes in the task association graph are traversed, the connected regions between the nodes are identified, the number of functional units covered by each connected region is calculated, and the connected region with the number of functional units exceeding a preset scale threshold is divided into the functional range of the current execution task to obtain the task context feature. Calculate the semantic similarity of the development stage semantic feature and each scene mode in the pre-defined scene mode library, extract the coverage of the functional range in the task context feature and convert it into a weight coefficient, weight the semantic similarity using the weight coefficient, and determine the scene state identifier of the current development scene.

[0008] Based on the scene state identifier, a semantic perception feature extractor is constructed, and the semantic perception feature extractor is used for syntax structure analysis and semantic dependency modeling of the code editing operation sequence, and the abstract syntax tree structure and program execution path graph of the code fragment are extracted. According to the scene state identifier, a syntax analysis rule set and a semantic modeling strategy corresponding to the current development scene are determined, the connection weight and the activation function parameter of the connection layer are configured based on the syntax analysis rule set and the semantic modeling strategy, and a semantic perception feature extractor is constructed. The semantic perception feature extractor is used for lexical analysis of the code fragment in the code editing operation sequence, identifiers, keywords and operators in the code fragment are identified and converted into a lexical unit sequence; the syntax structure of the code fragment is analyzed, the syntax tree nodes are recursively constructed according to the syntax rules of the programming language, the parent-child relationship and sibling relationship between the syntax tree nodes are established, and the abstract syntax tree structure of the code fragment is generated. Traverse the syntax tree nodes in the abstract syntax tree structure, identify the variable definition position and the variable reference position corresponding to the syntax tree nodes, establish the data flow dependency relationship chain between the variable definition position and the variable reference position, identify the function call position and the function definition position corresponding to the syntax tree nodes, establish the control flow jump relationship chain between the function call position and the function definition position, and combine the data flow dependency relationship chain and the control flow jump relationship chain to form the program execution path graph.

[0009] The code structured features are subjected to code pattern matching and abnormal pattern positioning to identify target function unit locations and potential defect locations in the code, and obtain a target code segment detection result including: The occurrence frequencies of the combination patterns in the code structured features are counted, and combination patterns with occurrence frequencies exceeding a preset frequency threshold are screened as high-frequency combination patterns; according to the data flow dependency relationship chains and the control flow jump relationship chains in the program execution path graph, a topologically connected subgraph with complete input-output closed loops is identified; and an overlapping region after spatial position alignment of the high-frequency combination patterns and the topologically connected subgraph is found, and the overlapping region that simultaneously satisfies the syntax completeness condition and the execution path closure condition is marked as the target function unit location; Syntax rule consistency checking is performed on the abstract syntax tree structure, syntax deviation degrees between nodes that violate syntax constraints and surrounding normal nodes are calculated to identify syntax abnormal nodes; data flow completeness checking is performed on the program execution path graph to identify abnormal influence ranges of the data flow dependency relationship chains with undefined variable references and the control flow jump relationship chains with unreachable paths, and code regions covered by the abnormal influence ranges are marked as execution abnormal regions; and code locations that simultaneously contain syntax abnormal nodes and execution abnormal regions are taken as the potential defect locations; The target function unit locations and the potential defect locations are integrated to generate the target code segment detection result.

[0010] Based on the target code segment detection result, operation time sequence features associated with the target code segment are extracted from the interface interaction event sequence and the file access record sequence, the operation time sequence features and the code structured features are subjected to cross-modal feature alignment to obtain a unified behavior vector including: Code line number ranges corresponding to the target function unit locations and the potential defect locations are extracted from the target code segment detection result, and time sequence association indexes of the code line number ranges and the interface interaction event sequence and the file access record sequence are established; According to the time sequence association indexes, interface interaction events and file access records with time stamps falling within editing time intervals corresponding to the code line number ranges are screened, control types and interaction action types of the interface interaction events are extracted, file paths and access operation types of the file access records are extracted, and operation time sequence features are constructed in chronological order; Node types and node level depths in the abstract syntax tree structure in the code structured features are extracted and encoded into syntax structure vectors, and start nodes and end nodes of data flow dependency relationship chains and control flow jump relationship chains in the program execution path graph are encoded into execution path vectors; After the operation timing features, the syntax structure vectors and the execution path vectors are cross-modality aligned, projected to a unified feature space and spliced and fused, the unified behavior vector is obtained.

[0011] The unified behavior vector is subjected to timing dependency reasoning and behavior pattern classification, and the coding behavior type of the developer is identified. The timing dependency relationship between the operation timing features, the syntax structure vectors and the execution path vectors in the unified behavior vector is constructed, and the timing dependency representation vector of the unified behavior vector in the time dimension is extracted according to the timing dependency relationship. The timing dependency representation vector is subjected to time window segmentation, the feature change rate of the timing dependency representation vector in each time window segment is calculated, and the time window segment with the feature change rate exceeding a preset change threshold is identified as a behavior transition node. In the timing dependency representation vector segment corresponding to the behavior transition node, the coordination degree between the operation timing feature change mode and the code structural feature change mode is calculated, and a behavior transition feature set is constructed. The mapping relationship between the timing dependency representation vector and the predefined coding behavior type is learned by using a behavior pattern classification network, the timing dependency representation vector is mapped to initial classification features according to the mapping relationship, the initial classification features are subjected to feature enhancement by using the behavior transition feature set, the similarity scores of the initial classification features after feature enhancement and each predefined coding behavior type are calculated, and the predefined coding behavior type with the highest similarity score is selected as the coding behavior type.

[0012] In a second aspect of the embodiment of the application, a multi-scene target detection and behavior recognition system based on deep learning is provided, which comprises: A first unit is configured to acquire a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and combine them into a multi-modal development behavior data stream. A second unit is configured to identify a development scene of the multi-modal development behavior data stream, and obtain a scene state identifier by analyzing development stage semantic features and task context features. A third unit is configured to construct a semantic perception feature extractor based on the scene state identifier, perform syntax structure analysis and semantic dependency modeling on the code editing operation sequence by using the semantic perception feature extractor, extract an abstract syntax tree structure and a program execution path graph of a code segment, and combine them to obtain code structural features. The code structural features are subjected to code pattern matching and abnormal pattern positioning, the location of a target function unit and the location of a potential defect in the code are identified, and a target code segment detection result is obtained. A fourth unit is configured to extract operation timing features associated with the target code segment from the interface interaction event sequence and the file access record sequence based on the target code segment detection result, perform cross-modal feature alignment between the operation timing features and the code structural features to obtain a unified behavior vector, perform timing dependency reasoning and behavior pattern classification on the unified behavior vector to identify the coding behavior type of the developer, and obtain a behavior recognition result.

[0013] A third aspect of the embodiment of the application, An electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; The processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0014] A fourth aspect of the embodiment of the application, A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0015] The beneficial effects of the present application are as follows: The present application can comprehensively capture multi-dimensional behavior information of the developer in the programming process by obtaining the code editing operation sequence, the interface interaction event sequence and the file access record sequence in the software development environment and forming a multi-modal development behavior data stream, thereby breaking through the limitations of traditional single data source analysis, providing a rich and complete data basis for subsequent scene recognition and behavior analysis, and significantly improving the accuracy and comprehensiveness of development behavior understanding.

[0016] The present application can accurately identify the location of the target function unit and the location of the potential defects by performing development scene recognition on the multi-modal development behavior data stream, analyzing the development stage semantic features and task context features to obtain a scene state identifier, and constructing a semantic perception feature extractor based on the scene state identifier to perform syntax structure analysis and semantic dependency modeling, thereby realizing an intelligent analysis process from scene perception to target detection, effectively improving the accuracy of code quality analysis and the efficiency of defect identification, and providing a more intelligent code review assistance capability for the developer. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a multi-scene target detection and behavior recognition method based on deep learning according to an embodiment of the application is shown in the figure. Figure 2 A flowchart of constructing a multi-modal development behavior data stream is shown in the figure. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0019] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments may not be described again for the same or similar concepts or processes.

[0020] Figure 1 The flowchart of the method for multi-scene target detection and behavior recognition based on deep learning of the embodiments of the present application is shown in FIG. 1, which comprises the following steps. Figure 1 obtaining a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and composing a multi-modal development behavior data stream; performing development scene recognition on the multi-modal development behavior data stream, obtaining a scene state identifier by analyzing development stage semantic features and task context features; constructing a semantic perception feature extractor based on the scene state identifier, performing syntax structure analysis and semantic dependency modeling on the code editing operation sequence by using the semantic perception feature extractor, extracting abstract syntax tree structures and program execution path graphs of code segments, and combining to obtain code structured features; performing code pattern matching and abnormal pattern positioning on the code structured features, identifying target function unit positions and potential defect positions in the code, and obtaining target code segment detection results; extracting operation timing features associated with the target code segment from the interface interaction event sequence and the file access record sequence based on the target code segment detection results, performing cross-modal feature alignment on the operation timing features and the code structured features to obtain unified behavior vectors; performing timing dependency reasoning and behavior pattern classification on the unified behavior vectors to identify the coding behavior types of the developers, and obtaining behavior recognition results.

[0021] In an optional implementation, obtaining a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and composing a multi-modal development behavior data stream comprises: In the software development environment, a code editor interface is monitored in real time, code editing operation events triggered by the developer are captured, operation type identifiers and operation position information in the code editing operation events are analyzed, and the code editing operation sequence is arranged in time sequence; ​The graphical interface layer is intercepted for events, and the interaction trigger records of the interface controls are obtained. The control identifiers and interaction action types in the interaction trigger records are extracted and arranged in chronological order to form the interface interaction event sequence. The file system access interface is monitored to record file read and write operations. The file path identifier and access timestamp corresponding to the read and write operations are extracted and arranged in chronological order to form the file access record sequence. The code editing operation sequence, the interface interaction event sequence, and the file access record sequence are aligned according to a unified time base to construct a cross-modal time index structure. Based on the cross-modal time index structure, the operation records that occur in the same time window in the three sequences are associated and bound to form the multimodal development behavior data stream.

[0022] like Figure 2 As shown, the method includes: During the implementation of the software development environment, an event listener component is deployed within the code editor module of the integrated development environment to achieve real-time capture of code editing operations. This listener component intercepts all operation events that modify the document content by registering with the editor's document model interface. For example, when a developer inputs text in the editor, the listener captures the insert event, parses the operation type identifier as INSERT, and obtains the operation position information, including line number 42, column number 15, and the inserted content as a variable declaration statement. When the developer performs a delete operation, the listener identifies the operation type identifier as DELETE, records the deletion start position as row 38, column 8, the deletion end position as row 38, column 23, and the length of the deleted content as 15 characters. Each captured editing operation event carries a timestamp accurate to the millisecond level; for example, the timestamp of an insert operation is 1678234567890, provided by the system clock. All captured editing operation events are stored in a linked list data structure in ascending order of timestamps, forming a code editing operation sequence. Each node in the sequence contains four fields: operation type, operation location, operation content, and timestamp.

[0023] In the event interception implementation of the graphical interface layer, all interface interaction events are captured by inserting an interception processor in the event dispatch mechanism of the interface framework, which is located at the top of the event dispatch link and can receive all events sent to the interface controls in priority. For example, when the developer clicks the compile button in the toolbar, the interception processor captures the mouse click event, extracts the control identifier as ToolbarButton_Compile, the interaction action type as CLICK, and records the event trigger timestamp as 1678234568123. When the developer expands a certain folder node in the project tree control, the interception processor identifies the control identifier as ProjectTreeNode_SrcFolder, the interaction action type as EXPAND, and records the timestamp as 1678234569456. For keyboard shortcut operations, the interception processor captures the keyboard event and parses the key combination as Ctrl plus S key, which is mapped to the save document interaction action, the control identifier as GlobalShortcut_Save, and the timestamp as 1678234570789. All interface interaction events are written into a queue structure in timestamp order to form an interface interaction event sequence. This sequence records the complete interaction trajectory of the developer with the development environment interface.

[0024] For the monitoring implementation of file system access, a hook function is deployed at the file system access interface layer to intercept all file operation calls. The hook function replaces the system's original file read-write function pointer, so that all file access requests are processed by the monitoring module before being forwarded to the underlying file system. For example, when the editor module requests to read the source code file, the hook function captures the read operation request, extracts the file path identifier as project / src / main / Controller.java, the access timestamp as 1678234571234, the operation type as READ, and the read data volume as 2048 bytes. When the code compilation module writes the compilation result to the output directory, the hook function records the write operation, the file path identifier as project / build / output / Main.class, the access timestamp as 1678234572567, the operation type as WRITE, and the write data volume as 4096 bytes. For file move or rename operations, the monitoring module records the source and target paths and marks the operation type as MOVE with the timestamp as 1678234573890. All file access records are stored in a log file in timestamp increasing order to form a file access record sequence.

[0025] After obtaining three independent sequences, they need to be time-aligned. Since the three monitoring modules run in different threads or processes, the timestamps maintained by each module have slight deviations. By recording a unified time reference point at system startup, each monitoring module calculates the offset relative to this reference point when recording the timestamp, ensuring the consistency of the timestamp. The cross-modal time index structure is established by using the time axis segmentation index method. The timeline of the entire development process is divided into fixed-length time windows, and the width of each time window is set to 500 milliseconds. For a code insertion operation with a timestamp of 1678234567890, the corresponding time window index is calculated by dividing the timestamp by 500, and the window index value is 3356469135. Record the operation under the entry corresponding to the window index. Similarly, the same window index calculation and registration process is performed for interface interaction events and file access records.

[0026] When performing association binding within the time window, traverse each time window index node, extract all operation records within the window, for example, the window index 3356469135 contains a code insertion operation, a toolbar click event and a file read record. When establishing the association relationship, a binding object is created, which contains reference pointers to the operation records in the three sequences. For a specific association instance, the record pointer in the code editing operation sequence points to the 127th insertion operation, the record pointer in the interface interaction event sequence points to the 89th click event, and the record pointer in the file access record sequence points to the 56th read operation. The association strength value is also recorded in the binding object, which is calculated according to the dispersion degree of the three operation timestamps. The closer the timestamps are, the higher the association strength is. All binding objects are organized into a streaming data structure according to the time window order, which is a multi-modal development behavior data stream. This data stream can completely present the association relationship between the code editing behavior, interface operation behavior and file access behavior of the developer in a certain time period, providing a data basis for subsequent development behavior analysis. The complete information carried by each node in the data stream includes time window identifier, window start timestamp, window end timestamp, code operation list, interface event list, file access list and cross-modal association relationship set.

[0027] In an optional implementation, the multi-modal development behavior data stream is subjected to development scene recognition, and the scene state identifier is obtained by analyzing the development stage semantic features and task context features, including: extract operation type distribution information and operation frequency change information from the multi-modal development behavior data stream, count the occurrence times of each operation type in the operation type distribution information, calculate the change rate of the operation frequency change information in the time dimension, determine the operation type with the highest occurrence time as the dominant operation type, and combine the dominant operation type and the change rate to obtain the development stage semantic feature; According to the control calling relationship and file access path dependency relationship in the multi-modal development behavior data stream, a task association graph is constructed, the nodes in the task association graph are traversed, the connected regions between the nodes are identified, the number of functional units covered by each connected region is calculated, and the connected region with a number of functional units exceeding a preset scale threshold is determined as the functional range of the current execution task to obtain the task context feature. The semantic similarity of the development stage semantic feature and each scene mode in the pre-defined scene mode library is calculated, the functional range coverage in the task context feature is extracted and converted into a weight coefficient, the semantic similarity is weighted using the weight coefficient, and the scene state identifier of the current development scene is determined.

[0028] When performing development scene recognition on a multi-modal development behavior data stream, real-time data stream from an integrated development environment is received, which includes code editing events, debugger interaction events, compiler calling events, version control operation events, and interface control clicking events. In actual application, a developer generates 328 keyboard input events, 15 file saving events, 8 debugging breakpoint setting events, 23 code jumping events, and 5 compilation execution events in a 1-minute time window. From these original event streams, operation type distribution information is extracted, and all events are classified into five main operation types: code writing class, debugging running class, file management class, configuration modification class, and document review class.

[0029] For the statistical process of operation type distribution information, an operation type counter array is established, all operation events in the time window are traversed, and the value of the corresponding counter is incremented according to the operation type label of the event. In the aforementioned case, the occurrence times of code writing class operation, debugging running class operation, file management class operation, configuration modification class operation, and document review class operation are 328, 31, 15, 3, and 2, respectively. By comparing the values of all counters, the operation type corresponding to the counter with the largest value is identified, and the code writing class operation is determined as the dominant operation type.

[0030] The calculation of operation frequency change information involves a sliding window analysis in the time dimension. The original data stream is divided into multiple sub-time windows according to a time interval of 30 seconds, and the total number of operations in each sub-time window is counted. In a specific case, the number of operations in four consecutive time windows is 45, 62, 89, and 132, respectively. The difference between the number of operations in adjacent time windows is calculated, resulting in a difference sequence of 17, 27, and 43. Each difference is divided by the number of operations in the previous time window to obtain a change rate sequence of 0.378, 0.435, and 0.483. The average value of the change rate sequence, 0.432, is taken as the final result of the operation frequency change rate. The dominant operation type and the change rate are combined to form the semantic feature of the development stage, which is represented as "code writing type operation-0.432 change rate" in this case.

[0031] The developer is monitored to have invoked the autocomplete control, syntax checking control, code formatting control, and refactoring tool control in the code editor, as well as accessed the core business logic file, data model definition file, interface configuration file, unit test file, and tool class library file. Each control and file is abstracted as a node in the task association graph, and a directed edge is established between two nodes when there is a call relationship in time or a reference dependency relationship in file structure. In the actual constructed task association graph, the core business logic file node and the data model definition file node have a reference edge, the code editor node and the autocomplete control node have a call edge, and the unit test file node and the core business logic file node have a dependency edge.

[0032] A graph traversal algorithm is used to traverse the task association graph to identify connected regions. Starting from any unvisited node, all reachable nodes are visited along the connection relationship of the edges, and these nodes are marked as the same connected region. In a certain task association graph, three connected regions are identified, the first connected region contains the core business logic file, data model definition file, interface configuration file and related editing control, a total of 12 nodes; the second connected region contains the unit test file, test tool control and debugger control, a total of 7 nodes; the third connected region contains the document review file and help system control, a total of 3 nodes.

[0033] The calculation of the number of functional units is based on the classification statistics of node types, which divides the nodes into source code file type functional units, configuration file type functional units, test file type functional units and tool control type functional units. The first connected region contains 5 source code file type functional units, 2 configuration file type functional units and 5 tool control type functional units, a total of 12 functional units. A preset scale threshold of 10 functional units is set, and the number of functional units of each connected region is compared with the threshold, and the first connected region is determined as the functional range of the current execution task, forming the task context feature.

[0034] The calculation of semantic similarity requires the support of a pre-defined scenario mode library, which includes code implementation scenario mode, function debugging scenario mode, requirement analysis scenario mode, code refactoring scenario mode, and document writing scenario mode. Each scenario mode contains typical dominant operation types and operation frequency variation rate range. The dominant operation type of the code implementation scenario mode is code writing type, and the variation rate range is 0.3 to 0.6. The dominant operation type in the semantic feature of the current development stage is matched with the dominant operation type of each scenario mode. When the matching is successful, a similarity base score of 0.7 is assigned, and when the matching is not successful, a similarity base score of 0.1 is assigned. The current variation rate 0.432 is compared with the variation rate range of the scenario mode. When it is within the range, the base score is increased by 0.2, and the semantic similarity of the code implementation scenario mode is 0.9.

[0035] The type diversity of the functional unit in the task context feature is quantified to obtain the functional range coverage. It is identified that the first connected region covers three types of source code file class, configuration file class and tool control class, and the type diversity index is 3. Divide the type diversity index by the preset maximum type number 5 to obtain the functional range coverage of 0.6. The process of converting the coverage value into a weight coefficient adopts a linear mapping relationship, which maps 0.6 to a weight coefficient of 1.2. The weight coefficient is used for weighted calculation of the semantic similarity. The semantic similarity of the code implementation scenario mode 0.9 is multiplied by the weight coefficient 1.2 to obtain the weighted similarity 1.08. Compare the weighted similarities of all scenario modes, select the scenario mode with the highest value as the current development scenario, and generate the scenario state identifier as the code implementation scenario. The identifier is used for subsequent intelligent auxiliary decision and resource scheduling optimization.

[0036] In an optional implementation, a semantic perception feature extractor is constructed based on the scenario state identifier, and the semantic perception feature extractor is used for syntax structure analysis and semantic dependency modeling of the code editing operation sequence to extract the abstract syntax tree structure and program execution path graph of the code fragment, including: The syntax analysis rule set and the semantic modeling strategy corresponding to the current development scenario are determined according to the scenario state identifier, the connection weight and the activation function parameter of the connection layer are configured based on the syntax analysis rule set and the semantic modeling strategy, and the semantic perception feature extractor is constructed; The semantic perception feature extractor is used for lexical analysis of the code fragment in the code editing operation sequence, and the identifiers, keywords and operators in the code fragment are identified and converted into a lexical unit sequence. The syntax structure of the lexical unit sequence is analyzed, the syntax tree nodes are recursively constructed according to the syntax rules of the programming language, the parent-child relationship and sibling relationship between the syntax tree nodes are established, and the abstract syntax tree structure of the code fragment is generated. Traverse the syntax tree nodes in the abstract syntax tree structure, identify the variable definition position and variable reference position corresponding to the syntax tree nodes, establish the data flow dependency relationship chain between the variable definition position and the variable reference position, identify the function call position and the function definition position corresponding to the syntax tree nodes, establish the control flow jump relationship chain between the function call position and the function definition position, and combine the data flow dependency relationship chain and the control flow jump relationship chain to form the program execution path graph.

[0037] Read the syntax analysis rule set corresponding to the scene state identifier from the configuration storage module. When the scene state identifier is "WebFrontend", load the JavaScript syntax rule set, which contains thirty-two types of syntax rule items such as variable declaration rules, function expression rules, arrow function rules, and asynchronous syntax rules. When the scene state identifier is "BackendService", load the Java syntax rule set, which contains forty-five types of syntax rule items such as class definition rules, interface declaration rules, generic parameter rules, and annotation processing rules. The semantic modeling strategy determines the semantic feature types that need to be focused on according to the scene state identifier. For the front-end development scene, the weight configuration of DOM operation semantics and event binding semantics is strengthened. For the back-end service scene, the weight configuration of database operation semantics and network request semantics is strengthened.

[0038] In the process of constructing the semantic-aware feature extractor, the parameters of the neural network connection layer are configured according to the loaded syntax analysis rule set. The feature extractor includes three processing levels: lexical analysis layer, syntax analysis layer, and semantic modeling layer. The connection weight matrix dimension of the lexical analysis layer is set to 512x256, which maps the input code character sequence to a lexical feature vector space. The connection weight matrix dimension of the syntax analysis layer is set to 256x128, which converts the lexical feature vector to a syntax structure feature. The connection weight matrix dimension of the semantic modeling layer is set to 128x64, which extracts high-level semantic representations. The activation function parameters are adjusted according to the semantic modeling strategy. For the semantic types that need to be strengthened, a smaller activation threshold of 0.3 is set. For the secondary semantic types, a larger activation threshold of 0.7 is set.

[0039] The constructed semantic-aware feature extractor is used to process the code editing operation sequence, and a code snippet "function calculatePrice(items) { let total = 0; for(let item of items) { total += item.price;} return total;}" is extracted from the code editing operation sequence. The lexical analysis module scans the code snippet, reads and identifies lexical units character by character from left to right. Five keyword type lexical units function, let, for, of, and return are identified, five identifier type lexical units calculatePrice, items, total, item, and price are identified, three operator type lexical units equal sign, plus equal sign, and dot are identified, and several separator type lexical units curly braces, round braces, and semicolons are identified. Each lexical unit is converted into a structured data object, including a lexical unit type field, a lexical unit text field, a start position field, and an end position field. The generated lexical unit sequence is arranged in the order of appearance in the source code, forming a lexical unit sequence with a length of 28.

[0040] A root node is created from the sequence start position. After detecting the function keyword, a function declaration node is created as a child of the root node, and the node type attribute value of the function declaration node is "FunctionDeclaration". The function name calculatePrice is parsed and an identifier node is created and attached to the function declaration node, and the text attribute value of the identifier node is "calculatePrice". The parameter list is parsed and a parameter node is created, and the node type attribute value of the parameter node is "Parameter" and the text attribute value is "items". When parsing the function body content, a block statement node is created as a child of the function declaration node. A variable declaration node is created under the block statement node, corresponding to the code "let total = 0", and the variable declaration node contains an identifier child node total and a literal child node 0. A loop statement node is created corresponding to the for loop structure, and the loop statement node contains a loop variable node item, an iteration object node items, and a loop body node. An assignment expression node is created under the loop body node, corresponding to the code "total += item.price", and the assignment expression node contains a left operand node total, an operator node plus equal, and a right operand node item.price. A return statement node is created corresponding to the return statement, and the return statement node contains a return value node total. The parent-child relationship between the nodes is established, the parent node of the function declaration node is the root node, the parent node of the block statement node is the function declaration node, and the parent nodes of the variable declaration node, the loop statement node, and the return statement node are all the block statement node. The sibling relationship is established, and the variable declaration node, the loop statement node, and the return statement node are sibling nodes. The completed abstract syntax tree structure contains a total of seventeen syntax tree nodes.

[0041] The abstract syntax tree structure is traversed for semantic dependency analysis. Starting from the root node, a depth-first traversal is performed. When the variable declaration node "let total = 0" is visited, the definition location of the variable total is recorded as the first line and the fifth column. When the assignment expression node "total += item.price" is visited, the reference location of the variable total is identified as the third line and the ninth column. A data flow dependency chain entry is established in memory. The entry includes a source location field recording the definition location "1:5", a target location field recording the reference location "3:9", a dependency type field value of "DataFlow", and a variable name field value of "total". When the return statement node "return total" is visited, another reference location of the variable total is identified as the fifth line and the tenth column, and a new data flow dependency chain entry is created. For the variable item, a data flow dependency chain is established from the loop variable declaration location of the third line and the fifteenth column to the reference location of the third line and the twenty-first column in the loop body.

[0042] When identifying control flow jump relationships, it is analyzed whether there is a function call expression in the code snippet. If the code contains a "processOrder(orderData)" call statement, the function name processOrder and the call location of the eighth line and the third column are extracted when the function call node is visited. The function definition node is searched in the abstract syntax tree. If the "function processOrder(data)" definition is found in the same file, the definition location of the fifteenth line and the first column is recorded. A control flow jump relationship chain entry is created, including a source location field recording the call location "8:3", a target location field recording the definition location "15:1", a dependency type field value of "ControlFlow", and a function name field value of "processOrder". If the function definition is located in an external module, the import location of the function is found in the import declaration statement, and a control flow jump relationship is established from the call location to the import location.

[0043] All collected data flow dependency chain and control flow jump relationship chain are combined to build a program execution path graph, the program execution path graph is represented by a directed graph data structure, the nodes in the graph correspond to code positions, and the edges in the graph correspond to dependency relationships. Create a graph node object corresponding to each variable definition position and reference position, the node object includes a position identifier attribute, a node type attribute, and an associated code attribute. Create a graph edge object to connect related nodes, the edge object includes a starting node reference attribute, a terminating node reference attribute, an edge type attribute, and a weight attribute. For data flow dependency relationships, the weight value of the edge is set to the frequency of variable reference, and the total variable is referenced twice, so the weight value of the corresponding edge is 2. For control flow jump relationships, the weight value of the edge is set to the frequency of function call. The complete program execution path graph contains 23 nodes and 18 edges, which completely describes the execution logic and data dependency relationship of the code snippet, and provides a semantic basis for subsequent code completion recommendation.

[0044] In an optional implementation, the code structured features are subjected to code pattern matching and abnormal pattern positioning to identify target functional unit positions and potential defect positions in the code, and obtain a target code segment detection result, including: Statistical frequency of occurrence of combined patterns in the code structured features, and screen combined patterns with an occurrence frequency exceeding a preset frequency threshold as high-frequency combined patterns; according to the data flow dependency chain and the control flow jump relationship chain in the program execution path graph, identify a topologically connected subgraph with a complete input-output closed loop; find an overlapping area after spatial position alignment of the high-frequency combined patterns and the topologically connected subgraph, and mark the overlapping area that simultaneously satisfies the syntax integrity condition and the execution path closure condition as the target functional unit position; Perform syntax rule consistency checking on the abstract syntax tree structure, identify syntax abnormal nodes by calculating the syntax deviation degree between nodes violating syntax constraints and surrounding normal nodes, perform data flow integrity checking on the program execution path graph, identify abnormal influence ranges of the data flow dependency chain with undefined variable references and the control flow jump relationship chain with unreachable paths, and mark code regions covered by the abnormal influence ranges as execution abnormal regions; code positions that simultaneously contain syntax abnormal nodes and execution abnormal regions are taken as the potential defect positions; Integrate the target functional unit positions and the potential defect positions to generate the target code segment detection result.

[0045] After receiving the data containing the abstract syntax tree structure, the program execution path graph and the code structured features, the frequency of occurrence of each type of combination pattern in the code structured features is started to be counted. The combination pattern refers to a code fragment composed of multiple syntax nodes in a specific order and hierarchical relationship, such as a complete conditional judgment statement containing the combination of judgment nodes, comparison operator nodes, variable reference nodes and code block nodes. The entire code structured feature set is traversed, and a counter is established for each identified combination pattern to record the number of times the combination pattern occurs in the code. When all the code is scanned, the ratio of the number of occurrences of each combination pattern to the total number of code lines is taken as the frequency of occurrence of the combination pattern. The preset frequency threshold is set to 5%, that is, when the frequency of occurrence of a combination pattern in the code exceeds 5%, the combination pattern is screened as a high-frequency combination pattern. For example, in a program containing 2000 lines of code, a combination pattern composed of assignment statements, loop structures and conditional judgments occurs 120 times, with a frequency of 6%, which exceeds the preset frequency threshold, and is therefore marked as a high-frequency combination pattern.

[0046] For the data flow dependency relationship chain and the control flow jump relationship chain in the program execution path graph, the identification of the topologically connected subgraph is performed. The data flow dependency relationship chain describes the definition, use and transmission relationship of variables, and the control flow jump relationship chain describes the branching, looping and jumping logic of program execution. Starting from the entry node of the program execution path graph, depth-first traversal is performed along the data flow dependency relationship chain and the control flow jump relationship chain. In the traversal process, the input data source and the output data destination of each node are recorded, and the entry point and the exit point of the control flow are recorded. When a node set is found to satisfy the following conditions, it is identified as a topologically connected subgraph: all data flow dependencies in the node set can be satisfied within the set, that is, there is no dependence on undefined variables outside the set; the node set has a clear control flow entry and exit, forming a complete execution loop. For example, a loop body structure has a loop condition judgment node as its entry, and the definition and use of variables within the loop body are completely completed within the loop body. After the loop ends, there is a clear exit node, and such a structure constitutes a topologically connected subgraph.

[0047] The identified high-frequency combination pattern is spatially aligned with the topological connection subgraph. Since the high-frequency combination pattern is a code fragment identified from the syntax level, and the topological connection subgraph is a code fragment identified from the execution path level, the representation dimensions of the two are different. By establishing a mapping relationship between the code line number and the syntax tree node and the execution path node, a unified representation of the two in the code space is realized. For each high-frequency combination pattern, its corresponding code line number range is extracted; for each topological connection subgraph, its contained code line number range is also extracted. By comparing the intersection of the two code line number ranges, the overlapping region is identified. In a specific case, a high-frequency combination pattern covers the 100th to 150th lines of code, and a topological connection subgraph covers the 120th to 180th lines of code, and the overlapping region of the two is the 120th to 150th lines. Further test whether the overlapping region satisfies the syntax integrity condition and the execution path closure condition. The syntax integrity condition requires that the code in the overlapping region constitutes a complete syntax unit, and there is no truncated statement or code block. The execution path closure condition requires that the execution path corresponding to the overlapping region has a clear starting point and ending point, forming a closed execution logic. Only the overlapping region that satisfies both conditions is marked as the target functional unit position.

[0048] In the identification process of potential defect positions, the abstract syntax tree structure is subjected to syntax rule consistency test. Syntax rules define the legal combination mode and constraint conditions of various syntax elements in the code, such as function calls must provide matching parameter number and type with function definition, and variables must be declared and initialized before use. Each node of the abstract syntax tree is traversed to check whether the node conforms to the corresponding syntax rule. When a node is found to violate the syntax constraint, the syntax deviation degree between the node and the surrounding normal nodes is calculated. The calculation of the syntax deviation degree is based on the accumulation of node attribute differences, including node type difference, node depth difference, and sub-node number difference. For example, in a function call node, the definition requires 3 parameters but only 2 parameters are provided, and the syntax anomaly of parameter mismatch is identified. The surrounding normal nodes refer to the nodes adjacent to the abnormal node in the abstract syntax tree and conforming to the syntax rule. The difference value of the number of parameters of the abnormal node and the number of parameters of the normal node is calculated, combined with the node type and level information, to obtain the syntax deviation degree value of the node. Nodes with syntax deviation degree exceeding the set threshold are marked as syntax abnormal nodes.

[0049] The data flow integrity is checked for the program execution path graph, and the undefined variable reference and unreachable path problem are mainly identified. The data flow integrity checking requires that each variable must be defined and assigned before being referenced, and there is a reachable execution path between the definition and the reference. The definition-use chain of the variable is constructed, and the propagation path of each variable from the definition point to the use point is tracked. When a variable is found to have no corresponding definition point at the use point, or the execution path between the definition point and the use point is interrupted, the data flow dependency chain is marked as an abnormal chain. For example, in a conditional branch structure, a variable is only defined in the branch where the judgment condition is true, but is also referenced in the branch where the condition is false. In this case, the use of the variable has an undefined reference problem. Further analysis of the unreachable path problem, through control flow analysis, identifies those code paths that cannot be executed under any execution scenario. Such paths are caused by logically contradictory conditional judgments or loop conditions that are never satisfied. The control flow jump relationship chain with the unreachable path is marked as an abnormal chain, and the influence range of the abnormal chain is analyzed. The abnormal influence range refers to the set of all code nodes affected by the abnormal data flow or abnormal control flow. Through the transitivity analysis of the dependency relationship, the code regions corresponding to these nodes are marked as execution abnormal regions.

[0050] The syntax abnormal nodes and the execution abnormal regions are associated and analyzed. The execution abnormality is caused by the syntax abnormality, and the root cause of the execution abnormality is often the syntax error, so the two have a causal relationship. It is checked whether each syntax abnormal node is located in a certain execution abnormal region, and whether each execution abnormal region contains a syntax abnormal node. When a code location contains both a syntax abnormal node and an execution abnormal region, the location is determined as a potential defect location. For example, there is a parameter type mismatch syntax abnormal node at line 200 of the code, and the function call where the node is located causes the subsequent data flow dependency chain to have an undefined variable reference, forming an execution abnormal region covering lines 200 to 220. Therefore, line 200 is marked as a potential defect location.

[0051] After identifying the target functional unit location and the potential defect location, the results are integrated. The integration process establishes a unified code segment description structure, including the start line number, the end line number, the code segment type, the associated syntax tree node set, the associated execution path node set, and the detection confidence, etc. For the target functional unit location, the high-frequency combination pattern type and the integrity measurement value of the topological connection subgraph are recorded. For the potential defect location, the syntax abnormal type, the syntax deviation value, the execution abnormal type, and the abnormal influence range are recorded. All identified target code segments are sorted in the order of code line numbers, and a target code segment detection result dataset containing complete location information, type annotation, and detailed attributes is generated.

[0052] In an optional implementation, the operation time sequence features associated with the target code segment are extracted from the interface interaction event sequence and the file access record sequence based on the target code segment detection result, the operation time sequence features are cross-modal feature aligned with the code structured features, and a unified behavior vector is obtained, including: The code line number range corresponding to the target function unit position and the potential defect position is extracted from the target code segment detection result, and a time sequence association index of the code line number range and the interface interaction event sequence and the file access record sequence is established; According to the time sequence association index, interface interaction events and file access records with time stamps falling within the editing time interval corresponding to the code line number range are screened, the control type and the interaction action type of the interface interaction events are extracted, the file path and the access operation type of the file access records are extracted, and the operation time sequence features are constructed in chronological order; The node type and the node level depth of the abstract syntax tree structure in the code structured features are extracted and encoded into a syntax structure vector, and the start node and the end node of the data flow dependency relationship chain and the control flow jump relationship chain of the program execution path graph are extracted and encoded into an execution path vector; After cross-modal alignment of the operation time sequence features, the syntax structure vector and the execution path vector, projection to a unified feature space and splicing fusion are performed to obtain the unified behavior vector.

[0053] The specific code line number information of the target function unit position and the potential defect position is obtained from the target code segment detection result. In one code file instance, it is assumed that the detected target function unit starts at line 120 and ends at line 180, and the potential defect position is located at line 150 to line 165. By analyzing the structured output of the target code segment detection result, the file path, the start line number and the end line number triple information contained in each position marker are extracted. For the above example, the record file path is the data processing module file under the project source code directory, the function unit line number range is 120 to 180, and the defect position line number range is 150 to 165.

[0054] A mapping relationship between the code line number range and the editing time interval is established, the historical submission records of the code version control system are accessed, and the timestamp information of the latest modification of each code line number range is queried. For the code segment from line 120 to line 180, it is retrieved that the segment has occurred editing operations during 10:23:15 am to 10:45:38 am at a certain time point. By analyzing the difference comparison data in the version control log, it is determined that the editing time window of the defect position from line 150 to line 165 is from 10:32:08 am to 10:38:52 am. The time interval information is bound to the corresponding code line number range to form a time sequence association index data structure.

[0055] The interface interaction event sequence records all operation behaviors of the developer in the integrated development environment, and each record contains a timestamp, a control identifier, an interaction action type and the like. For the time window from 10:32:08 am to 10:38:52 am, all events with timestamps falling within the interval are extracted by traversing the entire event sequence. In this example, fifteen interface interaction events are filtered, including a text input action performed on the code editor control at 10:32:15 am, a button click action performed on the debugging toolbar control at 10:33:42 am, a right-click menu selection action performed on the variable monitoring window control at 10:35:18 am, a text deletion action performed on the code editor control at 10:36:05 am, and a tree node expansion action performed on the function navigation panel control at 10:37:30 am.

[0056] The file access record sequence captures all operations on the project file system during development, and each record contains a timestamp, a file full path, an access operation type and the like. For the same time window, eight related records are located in the file access record sequence, including a read operation performed on the parameter definition file in the configuration file directory at 10:32:20 am, an open operation performed on the auxiliary function file in the tool class library directory at 10:33:55 am, a write operation performed on the unit test file in the test case directory at 10:34:48 am, an append operation performed on the running log file in the log output directory at 10:36:12 am, and a read operation performed on the third-party module file in the dependency library directory at 10:37:05 am.

[0057] The interface interaction events and file access records obtained through screening are arranged and combined according to the chronological order of timestamps, a unified timeline is created, the two types of records are mixed and sorted, the text input action of the code editor is recorded at 10:32:15 am, the reading operation of the configuration parameter file is recorded at 10:32:20 am, the button click action of the debugging toolbar is recorded at 10:33:42 am, and the opening operation of the auxiliary function file is recorded at 10:33:55 am. The key feature fields of each record are extracted, the control type identifier and the interaction action type identifier are extracted for the interface interaction events, and the directory hierarchy of the file path and the access operation type identifier are extracted for the file access records. These features are organized into a vector sequence in chronological order to form the original representation of the operation timing features.

[0058] The abstract syntax tree expresses the syntax composition of the source code in a tree structure, each node represents a syntax unit, starting from the root node, visiting the program entry node in the first layer, the node type is marked as function definition node and the layer depth is 1, visiting the parameter declaration node in the second layer, the node type is marked as parameter list node and the layer depth is 2, visiting the statement block node in the third layer, the node type is marked as compound statement node and the layer depth is 3. Assign a unique integer code to each node type, encode the function definition node as 101, the parameter list node as 102, the compound statement node as 103, the conditional judgment node as 104, and the loop control node as 105. Combine the node type code with the node layer depth value to form the binary tuple representation of each node, map the binary tuple to a fixed-length dense vector through word embedding technology, and arrange all node vectors in the order of traversal to form the syntax structure vector.

[0059] The program execution path graph depicts the execution logic of the code in a directed graph structure, the nodes represent code statements, the edges represent execution flow, the data flow dependency chain describes the definition and use relationship of variables between different statements, for example, variable X is defined at node A and used at node B, then there is a data flow dependency edge from node A to node B. The control flow jump relationship chain describes the branching and looping logic of program execution, for example, there is a conditional judgment at node C, and according to the judgment result, it jumps to node D or node E, then there are control flow jump edges from node C to node D and node E respectively. Traverse all edges in the execution path graph, record the start node identifier and the end node identifier for each edge, record the variable name involved for the data flow dependency edge, and record the jump condition type for the control flow jump edge. The identifiers of the start node and the end node are converted into numerical indexes through hash mapping, combined with the type label of the edge to form a triple representation, and all triples are converted into dense vectors through an embedding layer and concatenated to form an execution path vector.

[0060] The operation timing feature, the syntax structure vector, and the execution path vector are projected into a unified feature space to realize cross-modal alignment. The operation timing feature is in the form of a time sequence, the syntax structure vector is in the form of a tree structure derived sequence, and the execution path vector is in the form of a graph structure derived sequence. The dimensions and representation spaces of the three are different. Three linear transformation layers are respectively constructed to transform the operation timing feature from the original 256-dimensional space to the 512-dimensional unified space, transform the syntax structure vector from the 384-dimensional space to the 512-dimensional unified space, and transform the execution path vector from the 320-dimensional space to the 512-dimensional unified space. The three transformed vector sequences have the same feature dimension, and are spliced along the sequence length dimension to splice the three vector sequences with lengths of 23, 45, and 37 into a fusion vector sequence with a length of 105. The fusion vector sequence is the unified behavior vector, which completely describes the multi-modal feature information of the code editing behavior.

[0061] In an optional implementation, the unified behavior vector is subjected to timing dependence reasoning and behavior pattern classification, and the type of the developer's coding behavior is identified. A timing dependence relationship between the operation timing feature, the syntax structure vector, and the execution path vector in the unified behavior vector is constructed, and a timing dependence representation vector of the unified behavior vector in the time dimension is extracted according to the timing dependence relationship. The timing dependence representation vector is segmented by a time window, a feature change rate of the timing dependence representation vector in each time window segment is calculated, and the time window segment with the feature change rate exceeding a preset change threshold is identified as a behavior transition node. In the timing dependence representation vector segment corresponding to the behavior transition node, a coordination degree between an operation timing feature change pattern and a code structured feature change pattern is calculated, and a behavior transition feature set is constructed. A mapping relationship between the timing dependence representation vector and a predefined coding behavior type is learned by using a behavior pattern classification network, the timing dependence representation vector is mapped to an initial classification feature according to the mapping relationship, the initial classification feature is subjected to feature enhancement by using the behavior transition feature set, a similarity score between the feature-enhanced initial classification feature and each predefined coding behavior type is calculated, and the predefined coding behavior type with the highest similarity score is selected as the coding behavior type.

[0062] The unified behavior vector includes operation timing features, syntax structure vectors, and execution path vectors. These three types of elements have inherent temporal dependency relationships. Operation timing features record every operation performed by the developer in the editor and the time at which it occurred, such as inserting a function declaration at time T1, adding a parameter list at time T2, and writing a function body at time T3. Syntax structure vectors reflect the evolution of the abstract syntax tree structure of the code under these operations, and execution path vectors capture the changing trajectory of the code logic flow. Aligning these three types of vectors on the time axis, a dependency matrix is constructed to express their dependency relationships. Each element of the dependency matrix represents the association strength between the operation features at a certain time and the corresponding syntax structure changes and execution path changes. By processing the dependency matrix through a deep recurrent network, the hidden layer state of the network gradually accumulates the dependency information in the time dimension, and the final output hidden state sequence is the temporal dependency representation vector, which has a dimension of 512 and a length consistent with the operation sequence length.

[0063] The temporal dependency representation vector is processed by time window segmentation, and a fixed window length of 30 seconds is set. In a specific case, the coding process of the developer from the 0th second to the 180th second is divided into 6 time window segments. The feature change rate of the temporal dependency representation vector in each window segment is calculated. Specifically, the representation vector at the start time of the window segment and the representation vector at the end time are extracted, the Euclidean distance between the two is calculated, and the feature change amount per unit time is obtained by dividing the window length. In this case, the feature change rate of the first window segment is 0.23, the second window segment is 0.19, the third window segment is 0.67, the fourth window segment is 0.21, the fifth window segment is 0.58, and the sixth window segment is 0.25. The preset change threshold is set to 0.45, so the third window segment and the fifth window segment are identified as behavior transition nodes, corresponding to the coding segments from time 60 to 90 seconds and from time 150 to 180 seconds, respectively.

[0064] At the identified behavior transition node, the operation timing feature sequence and the code structural feature sequence in the corresponding time period are extracted to calculate the synergy degree. The operation timing feature change pattern is described by analyzing the distribution change of operation types in the time period. For example, in the third window segment, the text deletion operation accounts for 70% in the first 10 seconds, and the text insertion operation accounts for 80% in the last 20 seconds. The dramatic change in the distribution of operation types constitutes the operation change pattern. The code structural feature change pattern focuses on the evolution of abstract syntax tree node types. In the same window segment, the syntax tree mainly deletes loop structure nodes and conditional judgment nodes in the first 10 seconds, and adds function call nodes and assignment statement nodes in the last 20 seconds. When calculating the synergy degree, the operation change pattern is vectorized into a 128-dimensional vector, and the structure change pattern is vectorized into a 128-dimensional vector. The cosine similarity of the two vectors is calculated to obtain the synergy degree value. In the third window segment, the synergy degree is 0.82, and in the fifth window segment, the synergy degree is 0.76. The synergy degree value, operation change pattern vector, structure change pattern vector, and window segment position information are combined to form a behavior transition feature set. Each behavior transition node corresponds to a 384-dimensional transition feature vector.

[0065] The behavior pattern classification network adopts a multi-layer perceptron architecture, including 3 hidden layers with 256, 128, and 64 neurons respectively. The network input is a complete time-dependent representation vector sequence, which is compressed into a fixed-length 512-dimensional vector through global average pooling operation. The network training stage uses a labeled data set, which contains five types of predefined coded behavior types: functional development behavior, code refactoring behavior, defect repair behavior, code optimization behavior, and document writing behavior. Each type of behavior contains 200 sample sequences, and the network is trained to learn the mapping relationship between time-dependent representation vectors and behavior types through supervised learning. After training, the last layer of the network outputs a 64-dimensional initial classification feature vector.

[0066] In the inference stage, the extracted behavior transition feature set is used to enhance the initial classification features. The specific enhancement method is to weight and average the transition feature vectors of all behavior transition nodes, and the weights are determined by the normalized synergy degree values of each transition node. In the aforementioned case, the weight of the third window segment is 0.52, and the weight of the fifth window segment is 0.48. After weighted averaging, a 384-dimensional transition feature comprehensive vector is obtained. The comprehensive vector is mapped to a 64-dimensional space through a linear transformation layer, and an element-level multiplication fusion operation is performed with the initial classification feature vector to obtain a 64-dimensional classification feature vector after feature enhancement.

[0067] The similarity scores between the enhanced classification feature vector and the class prototype vectors of five pre-defined coding behavior types are calculated, each class prototype vector is the center point of the classification feature vectors of all samples in the training set of the class, and is obtained by a clustering method. The similarity calculation adopts a cosine similarity measurement method, and the cosine values of the feature vector to be classified and the five class prototype vectors are calculated respectively. In this case, the similarity with the function development behavior is 0.43, the similarity with the code refactoring behavior is 0.81, the similarity with the defect repair behavior is 0.35, the similarity with the code optimization behavior is 0.29, and the similarity with the document writing behavior is 0.18. The code refactoring behavior with the highest similarity score is selected as the final behavior type classification result of the coding fragment, and the automatic recognition of the developer coding behavior is completed.

[0068] The embodiment of the application is a multi-scene target detection and behavior recognition system based on deep learning, and the system comprises: A first unit is configured to acquire a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and form a multi-modal development behavior data stream. A second unit is configured to perform development scene recognition on the multi-modal development behavior data stream, analyze development stage semantic features and task context features, and obtain a scene state identifier. A third unit is configured to construct a semantic perception feature extractor based on the scene state identifier, perform syntax structure analysis and semantic dependency modeling on the code editing operation sequence by using the semantic perception feature extractor, extract an abstract syntax tree structure and a program execution path graph of a code fragment, and combine to obtain code structured features. The code structured features are subjected to code pattern matching and abnormal pattern positioning to identify a target function unit position and a potential defect position in the code, and a target code segment detection result is obtained. A fourth unit is configured to extract operation timing features associated with the target code segment from the interface interaction event sequence and the file access record sequence based on the target code segment detection result, perform cross-modal feature alignment on the operation timing features and the code structured features to obtain a unified behavior vector, perform timing dependency reasoning and behavior pattern classification on the unified behavior vector to identify a coding behavior type of a developer, and obtain a behavior recognition result.

[0069] In a third aspect of the embodiment of the application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0070] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method described above.

[0071] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present application.

[0072] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scene target detection and behavior recognition method based on deep learning, characterized in that, The method comprises the following steps: acquiring a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and composing a multi-modal development behavior data stream; performing development scene recognition on the multi-modal development behavior data stream, obtaining a scene state identifier by analyzing development stage semantic features and task context features; constructing a semantic perception feature extractor based on the scene state identifier, performing syntax structure analysis and semantic dependency modeling on the code editing operation sequence by using the semantic perception feature extractor, extracting abstract syntax tree structures and program execution path graphs of code segments, and combining to obtain code structured features; performing code pattern matching and abnormal pattern positioning on the code structured features to identify target functional unit positions and potential defect positions in the code, and obtaining a target code segment detection result; extracting operation timing features associated with the target code segment from the interface interaction event sequence and the file access record sequence based on the target code segment detection result, performing cross-modal feature alignment on the operation timing features and the code structured features to obtain a unified behavior vector; performing timing dependency reasoning and behavior pattern classification on the unified behavior vector to identify the coding behavior type of the developer, and obtaining a behavior recognition result.

2. The method of claim 1, wherein, acquiring a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and composing a multi-modal development behavior data stream comprises: in a software development environment, real-time monitoring of a code editor interface, capturing code editing operation events triggered by the developer, analyzing the operation type identifier and operation position information in the code editing operation event, and arranging in time sequence to form the code editing operation sequence; intercepting events of the graphical interface layer, obtaining interaction trigger records of the interface controls, extracting the control identifier and interaction action type in the interaction trigger records, and arranging in time sequence to form the interface interaction event sequence; monitoring the file system access interface, recording the read operation and write operation of the file, extracting the file path identifier and access timestamp corresponding to the read operation and write operation, and arranging in time sequence to form the file access record sequence; aligning the code editing operation sequence, the interface interaction event sequence and the file access record sequence in time sequence according to a unified time reference, constructing a cross-modal time index structure, associating and binding the operation records occurring in the same time window in the three sequences based on the cross-modal time index structure, and forming the multi-modal development behavior data stream.

3. The method of claim 1, wherein, performing development scene recognition on the multi-modal development behavior data stream, obtaining a scene state identifier by analyzing development stage semantic features and task context features comprises: extracting operation type distribution information and operation frequency change information from the multi-modal development behavior data stream, counting the number of occurrences of each operation type in the operation type distribution information, calculating the change rate of the operation frequency change information in the time dimension, determining the operation type with the highest number of occurrences as the dominant operation type, and combining the dominant operation type and the change rate to obtain the development stage semantic features; According to the control call relationship and file access path dependency relationship in the multi-modal development behavior data flow, a task association graph is constructed, nodes in the task association graph are traversed, a connected region between nodes is identified, a number of functional units covered by each connected region is calculated, a connected region with a number of functional units exceeding a preset scale threshold is delimited as a functional range of a current execution task, and a task context feature is obtained; The semantic similarity of the development stage semantic feature and each scene mode in a predefined scene mode library is calculated, the functional range coverage in the task context feature is extracted and converted into a weight coefficient, the semantic similarity is weighted using the weight coefficient, and the scene state identifier of the current development scene is determined.

4. The method of claim 1, wherein, Based on the scene state identifier, a semantic perception feature extractor is constructed, the code editing operation sequence is parsed in syntax structure and modeled in semantics using the semantic perception feature extractor, and an abstract syntax tree structure and a program execution path graph of a code segment are extracted, including: According to the scene state identifier, a set of syntax parsing rules and a semantic modeling strategy corresponding to the current development scene are determined, connection weights and activation function parameters of a connection layer are configured based on the set of syntax parsing rules and the semantic modeling strategy, and a semantic perception feature extractor is constructed; The semantic perception feature extractor is used to perform lexical analysis on a code segment in the code editing operation sequence, identify identifiers, keywords, and operators in the code segment, and convert them into a lexical unit sequence; the lexical unit sequence is parsed in syntax structure, syntax tree nodes are recursively constructed according to the syntax rules of the programming language, parent-child relationships and sibling relationships between syntax tree nodes are established, and an abstract syntax tree structure of the code segment is generated; The syntax tree nodes in the abstract syntax tree structure are traversed, variable definition positions and variable reference positions corresponding to the syntax tree nodes are identified, a data flow dependency relationship chain between the variable definition positions and the variable reference positions is established, function call positions and function definition positions corresponding to the syntax tree nodes are identified, a control flow jump relationship chain between the function call positions and the function definition positions is established, and the data flow dependency relationship chain and the control flow jump relationship chain are combined to form the program execution path graph.

5. The method of claim 4, wherein, The code structured feature is matched with a code pattern and positioned in an abnormal pattern, target functional unit positions and potential defect positions in the code are identified, and a target code segment detection result is obtained, including: The occurrence frequency of a combination mode in the code structured feature is counted, the combination mode with an occurrence frequency exceeding a preset frequency threshold is selected as a high-frequency combination mode, a topological connection subgraph with a complete input-output closed loop is identified according to the data flow dependency relationship chain and the control flow jump relationship chain in the program execution path graph, and an overlapping region after spatial position alignment between the high-frequency combination mode and the topological connection subgraph is found. The overlapping region that simultaneously satisfies the syntax integrity condition and the execution path closure condition is marked as the target functional unit position. The abstract syntax tree structure is subjected to syntax rule consistency test, syntax deviation degrees between nodes violating syntax constraints and surrounding normal nodes are calculated, and syntax abnormal nodes are identified; the program execution path graph is subjected to data flow integrity test, abnormal influence ranges of data flow dependency chains with undefined variable references and control flow jump relationship chains with unreachable paths are identified, and code regions covered by the abnormal influence ranges are marked as execution abnormal regions; code positions containing both syntax abnormal nodes and execution abnormal regions are taken as the potential defect positions; The target code segment detection result is generated by integrating the target function unit positions and the potential defect positions.

6. The method of claim 1, wherein, Based on the target code segment detection result, operation time sequence features associated with the target code segment are extracted from the interface interaction event sequence and the file access record sequence, the operation time sequence features and the code structured features are subjected to cross-modal feature alignment, and a unified behavior vector is obtained, including: Code line number ranges corresponding to the target function unit positions and the potential defect positions are extracted from the target code segment detection result, and time sequence association indexes of the code line number ranges and the interface interaction event sequence and the file access record sequence are established; According to the time sequence association indexes, interface interaction events and file access records with time stamps falling within editing time intervals corresponding to the code line number ranges are screened, control type and interaction action type of the interface interaction events are extracted, file path and access operation type of the file access records are extracted, and operation time sequence features are constructed in chronological order; Node type and node level depth of the abstract syntax tree structure in the code structured features are extracted and encoded as syntax structure vectors, and start node and end node of data flow dependency relationship chains and control flow jump relationship chains of the program execution path graph are extracted and encoded as execution path vectors; After cross-modal alignment of the operation time sequence features, the syntax structure vectors and the execution path vectors, projection to a unified feature space and splicing and fusion are performed, and the unified behavior vector is obtained.

7. The method of claim 1, wherein, The unified behavior vector is subjected to time sequence dependency reasoning and behavior pattern classification, and the developer's coding behavior type is identified, including: Time sequence dependency relationships between operation time sequence features, syntax structure vectors and execution path vectors in the unified behavior vector are constructed, and time sequence dependency representation vectors of the unified behavior vector in the time dimension are extracted according to the time sequence dependency relationships; The time sequence dependency representation vectors are subjected to time window segmentation, feature change rates of the time sequence dependency representation vectors in each time window segment are calculated, and the time window segments with the feature change rates exceeding a preset change threshold are identified as behavior conversion nodes; In the time sequence dependency representation vector segment corresponding to the behavior conversion node, the coordination degree between operation time sequence feature change patterns and code structured feature change patterns is calculated, and a behavior conversion feature set is constructed; The behavior pattern classification network learns a mapping relationship between a time-dependent representation vector and a predefined coded behavior type, maps the time-dependent representation vector to an initial classification feature according to the mapping relationship, performs feature enhancement on the initial classification feature by using the behavior conversion feature set, calculates a similarity score of the initial classification feature after feature enhancement and each predefined coded behavior type, and selects a predefined coded behavior type with the highest similarity score as the coded behavior type.

8. A deep learning based multi-scene object detection and behavior recognition system for implementing the method of any one of claims 1-7, characterized in that, Comprise: A first unit for obtaining a code editing operation sequence, an interface interaction event sequence and a file access record sequence in a software development environment and composing a multi-modal development behavior data stream; A second unit for performing development scene recognition on the multi-modal development behavior data stream, obtaining a scene state identifier by analyzing development stage semantic features and task context features; A third unit for constructing a semantic perception feature extractor based on the scene state identifier, performing syntax structure analysis and semantic dependency modeling on the code editing operation sequence by using the semantic perception feature extractor, extracting an abstract syntax tree structure and a program execution path graph of a code snippet, and combining to obtain code structured features; performing code pattern matching and abnormal pattern positioning on the code structured features, identifying a target function unit position and a potential defect position in the code, and obtaining a target code segment detection result; A fourth unit for extracting operation time sequence features associated with a target code segment from the interface interaction event sequence and the file access record sequence based on the target code segment detection result, performing cross-modal feature alignment on the operation time sequence features and the code structured features, obtaining a unified behavior vector; performing time-dependent reasoning and behavior pattern classification on the unified behavior vector, identifying a developer's coding behavior type, and obtaining a behavior recognition result.

9. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; Wherein, the processor is configured to call the instructions stored in the memory to execute the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 7. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 7.

Images