A method and device for implementing remote video mediation based on big data

Abstract

The present application provides a method and device for implementing remote video mediation based on big data, which extracts features from image samples based on a feature extraction component to obtain a sample feature vector, obtains an identified mediation object based on the sample feature vector and a mediation object recognition component, obtains an identified visual relationship based on the identified mediation object and a visual relationship detection component, and collaboratively debugs the feature extraction component, mediation object recognition component, and visual relationship detection component based on the identified mediation object, the identified visual relationship, and the prior annotation, optimizes the sample feature vector based on the optimized feature extraction component and the identified visual relationship, jumps to the step of identifying the mediation object of the image sample based on the sample feature vector and the mediation object recognition component for repeated debugging, and obtains each network component when the debugging cutoff requirement is met. Using the above process, the present application can ensure the accuracy of the mediation object recognition component and the visual relationship detection component.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for implementing remote video mediation based on big data. Background Art

[0002] With the rapid development of information technology and artificial intelligence technology, remote video mediation, as an efficient and convenient dispute resolution method, has gradually been widely used in many fields such as justice, administration, and commerce. However, in the process of remote video mediation, how to accurately and in real time identify the mediation objects and the visual relationship between them has become a technical problem that needs to be solved urgently. In recent years, with the continuous advancement of computer vision and machine learning technology, especially breakthroughs in the field of deep learning, new solutions have been provided for object recognition and visual relationship detection in remote video mediation. However, most of the existing object recognition and visual relationship detection methods are designed for general scenarios and lack special optimization for remote video mediation scenarios. When applied to remote video mediation, these methods often face problems such as low recognition accuracy, poor robustness, and insufficient real-time performance. Summary of the invention

[0003] In view of this, the present application provides a method for implementing remote video mediation based on big data. The technical solution of the present application is implemented as follows:

[0004] On the one hand, an embodiment of the present application provides a method for implementing remote video mediation based on big data, the method comprising: obtaining a remote mediation room video, and obtaining a target mediation room image in the remote mediation video; performing feature extraction on the target mediation room image based on a debugged target feature extraction component, and obtaining a feature vector corresponding to the target mediation room image; identifying the mediation object of the target mediation room image based on the feature vector and a debugged target mediation object recognition component, and obtaining an identified mediation object; identifying the visual relationship of the mediation object of the target mediation room image based on the identified mediation object and a debugged target visual relationship detection component, and obtaining an identified visual relationship; and marking in the target mediation room image based on the identified mediation object and the identified visual relationship. Among them, the debugging process of the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component includes: obtaining an image sample and a priori annotation corresponding to the image sample; the prior annotation includes the mediation object annotation corresponding to the image sample and the visual relationship annotation corresponding to the image sample; performing feature extraction on the image sample based on the feature extraction component to obtain a sample feature vector corresponding to the image sample; identifying the mediation object of the image sample according to the sample feature vector and the mediation object recognition component to obtain an identified mediation object; identifying the mediation object visual relationship of the image sample according to the identified mediation object and the visual relationship detection component to obtain an identified visual relationship; determining a first debugging cost according to an error between the identified mediation object and the mediation object annotation, determining a second debugging cost according to an error between the identified visual relationship and the visual relationship annotation, determining a first optimization direction according to the first debugging cost, determining a second optimization direction according to the second debugging cost, and fusing all The first optimization direction and the second optimization direction are combined to obtain a fused optimization direction, and the feature extraction component, the mediation object recognition component and the visual relationship detection component are collaboratively debugged according to the fused optimization direction to optimize the feature extraction component, the mediation object recognition component and the visual relationship detection component; the image sample is loaded into the optimized feature extraction component, and feature extraction is performed on the image sample based on the optimized feature extraction component to obtain a feature vector, and the identified visual relationship is integrated with the feature vector to optimize the sample feature vector; the step of jumping to the step of identifying the mediation object of the image sample according to the sample feature vector and the mediation object recognition component is repeatedly debugged, and when the debugging cutoff requirement is met, the debugged target feature extraction component, the target mediation object recognition component and the target visual relationship detection component are obtained, and the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component are used to jointly detect the visual relationship of the mediation object of the target mediation room image.

[0005] On the other hand, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above method when executing the program.

[0006] The technical effect of the present application is as follows: after acquiring the target mediation room image, the present application extracts features of the target mediation room image based on the debugged target feature extraction component, obtains the feature vector corresponding to the target mediation room image, identifies the mediation object of the target mediation room image based on the feature vector and the debugged target mediation object recognition component, obtains the identified mediation object, identifies the visual relationship of the mediation object of the target mediation room image based on the identified mediation object and the debugged target visual relationship detection component, obtains the identified visual relationship, and marks the target mediation room image based on the identified mediation object and the identified visual relationship, because the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component The results are obtained through repeated collaborative debugging based on image samples. Each debugging is based on the recognized visual relationship output by the visual relationship detection component obtained from the previous sample, and the sample feature vector input into the mediation object recognition component obtained from the previous sample is optimized. In the debugging phase, the spatial dependency between the mediation object recognition component and the visual relationship detection component is considered. The obtained mediation object recognition component and target visual relationship detection component have high robustness and universality, and can accurately obtain the mediation objects and visual relationships in the target mediation room image, making the labeling results based on the recognition of mediation objects and the recognition of visual relationships more accurate, ensuring the openness and transparency of the mediation process, and ensuring the real-time monitoring response. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of the implementation flow of a method for implementing remote video mediation based on big data provided in an embodiment of the present application.

[0008] Figure 2 A schematic diagram of the training process of each network component provided in an embodiment of the present application.

[0009] Figure 3 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The embodiment of the present application provides a method for implementing remote video mediation based on big data, which can be executed by a processor of a computer device, wherein the computer device can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, and a desktop computer.

[0011] Figure 1 A schematic diagram of an implementation flow of a method for implementing remote video mediation based on big data provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0012] Step S101: Acquire a remote mediation room video, and acquire a target mediation room image in the remote mediation video.

[0013] The mediation room video contains rich information such as the real-time interaction, expression changes, body movements, etc. between the two parties, which is crucial for subsequent analysis, recognition and annotation. Therefore, accurate and efficient acquisition of the target mediation room image is the basis for the smooth progress of subsequent steps.

[0014] Step S101 receives a mediation room video stream from a remote mediation device (such as a camera), and selects or captures a target mediation room image from the received video stream.

[0015] The computer device establishes a connection with the remote mediation device through the network and receives the mediation room video stream in real time. This involves, for example, the decoding process of video encoding, that is, restoring the compressed video data transmitted over the network to the original, playable video stream. For example, if the video is encoded in H.264, the computer uses the corresponding decoder to decode the video data into pixel data in YUV or RGB format. After receiving the video stream, a specific frame is selected or intercepted from it as the target mediation room image. This process can adopt different strategies according to actual needs: the computer device can set a fixed time interval (such as intercepting one frame per second) to automatically intercept images from the video stream. Alternatively, through a preset event detection mechanism (such as sound detection, motion detection, etc.), when a specific event is detected, the computer device automatically intercepts the current frame. For example, when a large movement is detected in the mediation room, it can be considered that the mediators are having an intense discussion at this time, so the frame is intercepted as the target mediation room image. Alternatively, in some cases, the mediator or device administrator wants to manually select a specific frame as the target mediation room image. At this time, the computer device can provide a user interface to allow the user to browse the video stream and manually mark the target frame.

[0016] Step S102: Acquire a target mediation room image, perform feature extraction on the target mediation room image based on the debugged target feature extraction component, and obtain a feature vector corresponding to the target mediation room image.

[0017] During remote video mediation, the target mediation room image contains interactive information of both parties, such as expressions, movements, and postures. This information is crucial for understanding the mediation process and analyzing the emotional state and behavior patterns of the mediation object. However, the original image data is huge and complex, and direct analysis is extremely difficult. Therefore, through feature extraction, the key information in the image is converted into a feature vector that can be processed by a computer. The feature vector is a high-dimensional data representation that can capture the key features in the image and provide a basis for subsequent analysis and recognition.

[0018] The execution process of step S102 mainly includes two parts: first, the computer device obtains the target mediation room image from step S101; second, the debugged target feature extraction component is used to extract features from the image to obtain a feature vector.

[0019] In step S101, the computer device has intercepted or selected the target mediation room image from the remote mediation room video. These images are stored in the form of a pixel matrix, for example, where each pixel represents a color value in the image. For example, a grayscale image can be represented as a two-dimensional array, where the value of each element represents the grayscale value of the corresponding pixel.

[0020] Feature extraction aims to extract useful feature information from raw image data. In remote video mediation scenarios, feature extraction components can capture key information such as the expression, movement, and posture of the mediation object. To achieve this goal, feature extraction components use complex machine learning models such as convolutional neural networks (CNNs) and deep belief networks (DBNs). During the feature extraction process, the computer device inputs the target mediation room image into the debugged feature extraction component. The component performs a series of complex calculations and processing on the image and finally outputs a feature vector. This feature vector is a high-dimensional data representation that can capture key features in the image.

[0021] For example, suppose that in a remote family dispute mediation scenario, the two parties mediate through video conferencing software. The computer device captures an image of the target mediation room containing the expressions and actions of the two parties from the mediation room video. In order to extract feature information from the image, the computer device uses a feature extraction component based on a convolutional neural network.

[0022] The convolutional neural network consists of multiple convolutional layers, pooling layers, and fully connected layers. In the feature extraction process, the image first passes through the convolutional layer for local feature extraction, then passes through the pooling layer for dimensionality reduction and feature selection, and finally passes through the fully connected layer to integrate the local features into a global feature vector. After multiple iterations and optimizations, the feature extraction component can accurately capture key information such as the expression and action of the mediation object and represent it in the form of a feature vector.

[0023] Convolutional neural network is a feasible image feature extraction model, which extracts local features in images by simulating the visual perception mechanism of the human eye. In CNN, the convolution layer extracts local features of the image by sliding convolution operations on the image through the convolution kernel; the pooling layer reduces the computational effort and overfitting risk by reducing the dimension and selecting features of local features; the fully connected layer integrates local features into a global feature vector.

[0024] The feature vector is the output of the feature extraction component. It is a high-dimensional data representation that can capture the key features of the image. In the remote video mediation scenario, the feature vector contains key information such as the expression, action, and posture of the mediation object. For example, a feature vector containing 1024 elements can be represented as: [0.12, 0.34, ..., 0.78], where the value of each element represents the importance or strength of the corresponding feature.

[0025] The performance of the feature extraction component depends on the quality of its design and implementation. In order to obtain better feature extraction results, the component is repeatedly debugged and optimized. This involves, for example, adjusting the network structure, selecting appropriate hyperparameters, using regularization techniques to prevent overfitting, etc. For example, during the debugging process, the performance of the feature extraction component can be optimized by changing the size and number of convolution kernels, adjusting the learning rate, adding Dropout layers, etc.

[0026] During the feature extraction process, each convolutional layer in the convolutional neural network will perform a convolution operation on the input image. Assuming the input image is X and the convolution kernel is W, the output Y of the convolution operation can be calculated by the following formula: ; Where M and N represent the number of rows and columns of the convolution kernel, respectively, and i and j represent the row and column indexes of the output feature map. Through multiple convolution operations and pooling operations, a feature vector containing key features can be finally obtained.

[0027] Step S103: identifying the mediation object of the target mediation room image according to the feature vector and the debugged target mediation object identification component to obtain the identified mediation object.

[0028] In remote video mediation scenarios, accurate identification of mediation objects is the basis for subsequent analysis, annotation, and intervention. Mediation objects may include mediators, parties, and observers. Their identities, behaviors, emotions, and other information are crucial for understanding the mediation process and evaluating the mediation effect. Therefore, step S103 provides key information for subsequent steps by identifying the mediation objects, ensuring the accuracy and effectiveness of the entire mediation process.

[0029] In step S102, the computer device has extracted a feature vector from the target mediation room image. This feature vector is a high-dimensional data representation that contains key information in the image, such as the contour, texture, color, etc. of the mediation object.

[0030] The target mediation object recognition component is a debugged and optimized machine learning model that can recognize the mediation object based on the input feature vector. This component can be, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN), or a traditional machine learning algorithm such as a support vector machine (SVM) or a random forest.

[0031] The computer device inputs the feature vector into the debugged target mediation object recognition component, which processes and analyzes the feature vector and finally outputs the recognition result. The recognition result includes, for example, the identity, location, quantity and other information of the mediation object. For example, if there are multiple people in the mediation room, the recognition component can accurately recognize the identity and location of each person.

[0032] For example, suppose that in a remote commercial dispute mediation scenario, there are multiple characters in the mediation room, such as a mediator, a plaintiff, a defendant, and an audience. The computer device first extracts a feature vector from the mediation room image in step S102. Then, in step S103, the computer device inputs the feature vector into the debugged target mediation object recognition component. This component is a deep learning model based on a convolutional neural network. After training and debugging with a large amount of data, it can accurately identify the identities of the characters in the mediation room. Finally, the recognition component outputs the identity and location information of each person in the mediation room, such as "mediator, location (100, 200)", "plaintiff, location (300, 400)", "defendant, location (500, 600)", etc.

[0033] The feature vector is the output result of step S102, and it contains the key information in the image. In step S103, the computer device may pre-process the feature vector, such as normalization, dimension reduction, etc., to improve the accuracy and efficiency of the recognition component.

[0034] The target mediation object recognition component is the core part of step S103. Its design and implementation take into account multiple factors, such as model selection, network structure design, hyperparameter setting, etc. For example, in a convolutional neural network, select appropriate convolution kernel size, step size, padding method and other parameters; in a fully connected layer, set the appropriate number of neurons and activation function, etc.

[0035] The output results of the recognition component include, for example, the identity, location, quantity, and other information of the mediation object. This information can be presented in the form of text, image annotation, data table, etc. For example, in image annotation, rectangular boxes of different colors can be used to mark the location and identity of different mediation objects.

[0036] The performance of the target mediation object recognition component depends on the quality of its design and implementation. In order to obtain better recognition results, the component is repeatedly debugged and optimized. This involves, for example, adjusting model parameters, increasing training data, and introducing regularization techniques to prevent overfitting. For example, during the training process, techniques such as cross-validation and early stopping can be used to evaluate the performance of the model and prevent overfitting.

[0037] Step S104: Identify the mediation object visual relationship of the target mediation room image according to the identified mediation object and the debugged target visual relationship detection component to obtain the identified visual relationship.

[0038] Step S104 further analyzes and identifies the visual relationships between the mediation objects in the target mediation room image by identifying the mediation objects and the debugged target visual relationship detection component. These visual relationships may represent the behavior, emotions, interaction patterns, etc. of the mediation objects, which are of great significance for in-depth understanding of the mediation process and evaluation of the mediation effect.

[0039] In remote video mediation scenarios, the visual relationship between mediation objects often contains rich information, such as their interaction patterns, emotional states, behavioral trends, etc. This information is crucial for a comprehensive understanding of the mediation process, timely discovery of potential problems, and effective intervention measures. Therefore, step S104 provides an important basis for subsequent analysis, annotation, and intervention by identifying the visual relationship between mediation objects, thereby ensuring the accuracy and effectiveness of the entire mediation process.

[0040] In step S103, the computer device has identified the mediation objects in the target mediation room image and obtained their identity, location and other information. This information is one of the inputs of step S104. The target visual relationship detection component is a debugged and optimized machine learning model, which can further analyze and identify the visual relationship between the mediation objects based on the input mediation object information. This component is, for example, a complex deep learning model, such as a convolutional neural network (GCN), a relation network, etc., or it may be a combination model based on a traditional machine learning algorithm. The computer device inputs the information for identifying the mediation object into the debugged target visual relationship detection component. The component processes and analyzes this information and finally outputs the recognition result. The recognition result includes, for example, information such as the type, strength, and direction of the visual relationship between the mediation objects. For example, in the mediation room, the mediator may be actively listening to the statements of the parties, and this interactive relationship can be identified as a "listening-statement" relationship.

[0041] For example, suppose that in a remote labor dispute mediation scenario, there are three people in the mediation room: a mediator, a plaintiff, and a defendant. In step S103, the computer device has identified the identities and locations of these three people. In step S104, the computer device inputs this information into the debugged target visual relationship detection component. This component is a model based on a graph convolutional neural network, which can comprehensively analyze the spatial relationship, actions, expressions, etc. between the mediation objects, and finally recognize that the mediator is actively listening to the plaintiff's statement, while the defendant appears to be more nervous. Therefore, the component outputs two visual relationships: "mediator-listening-plaintiff" and "defendant-nervous".

[0042] Before being input into the target visual relationship detection component, the information of the mediation object is appropriately represented. This includes, for example, the identity, position, action, expression and other features of the mediation object. These features can be represented in the form of vectors, matrices or graphs for subsequent processing and analysis.

[0043] The target visual relationship detection component is the core part of step S104. Its design and implementation take into account multiple factors, such as the choice of model, the design of network structure, the method of feature extraction and fusion, etc. For example, in a graph convolutional neural network, the mediation objects can be represented as nodes in the graph, and the visual relationship between them can be represented as edges. Through the graph convolution operation, the model can capture the spatial relationship and dependency between nodes, and then identify the visual relationship between mediation objects.

[0044] The output results of the target visual relationship detection component include, for example, information such as the type, strength, and direction of the visual relationship between the mediation objects. This information can be presented in the form of text, image annotation, data table, etc. For example, in image annotation, lines or arrows of different colors can be used to mark the type and direction of the visual relationship between different mediation objects. In addition, in order to enhance the interpretability of the results, the computer device can also provide additional explanatory information, such as the confidence of the visual relationship, relevant evidence, etc. The performance of the target visual relationship detection component depends on the quality of its design and implementation. In order to obtain better recognition results, the component is repeatedly debugged and optimized. This involves, for example, adjusting model parameters, increasing training data, introducing regularization technology to prevent overfitting, etc. For example, during the training process, cross-validation, early stopping and other techniques can be used to evaluate the performance of the model and prevent overfitting; at the same time, visualization technology can be used to observe the learning process and feature representation of the model in order to better understand and optimize the model.

[0045] Step S105: marking the target mediation room image according to the identified mediation objects and the identified visual relationships.

[0046] In the process of remote video mediation, accurate identification of mediation objects and their visual relationships provides an important basis for understanding mediation dynamics and evaluating mediation effects. However, these analysis results may be difficult for non-professionals to understand directly. Therefore, step S105 presents the analysis results in an intuitive and easy-to-understand manner by marking the target mediation room image, which helps mediators, parties and related personnel to better grasp the mediation process, identify problems in a timely manner and take corresponding measures.

[0047] In the previous step, the computer device has identified the mediation objects and their visual relationships in the target mediation room image, and this information is the input of step S105.

[0048] According to actual needs and application scenarios, the computer device can choose different annotation methods. Common annotation methods include text annotation, graphic annotation, color annotation, etc. For example, in text annotation, text descriptions can be added to the image to indicate the identity and visual relationship of the mediation object; in graphic annotation, different shapes and lines can be used to represent the mediation object and its visual relationship.

[0049] The computer device marks the target mediation room image according to the selected marking method. The marking process involves, for example, image processing and graphics drawing technology to ensure the accuracy and aesthetics of the marking.

[0050] After the annotation is completed, the computer device outputs the annotated image to a display device or storage medium for relevant personnel to view and analyze.

[0051] For example, suppose that in a remote family dispute mediation scenario, there are three people in the mediation room: a mediator, a plaintiff, and a defendant. In the previous steps, the computer device has identified the identities and positions of these three people, and identified the visual relationships between them, such as "mediator-listening-plaintiff" and "defendant-tense". In step S105, the computer device chooses to use a combination of text annotation and color annotation to annotate the target mediation room image. Specifically, it marks the positions of the mediator, plaintiff, and defendant with rectangular boxes of different colors on the image, and adds text next to the rectangular boxes to explain their identities; at the same time, it uses lines with arrows to indicate that the mediator is listening to the plaintiff's statement, and the line colors are distinguished according to the strength of the visual relationship. Finally, the annotated image clearly shows the key information in the mediation process, which helps relevant personnel better understand the mediation dynamics.

[0052] The annotation process involves, for example, image processing techniques, such as image scaling, rotation, and cropping, to ensure the accuracy and aesthetics of the annotation. For example, when annotating the position of a mediation object, the size and position of the rectangular box may be adjusted according to the actual size and resolution of the image. Graphic drawing techniques are also used in the annotation process to draw annotation elements, such as rectangular boxes, lines, text, etc. The size, color, font and other attributes of these elements can be customized according to actual needs. The choice of annotation method should be determined according to actual needs and application scenarios. For example, in a scene that displays the identity and visual relationship of the mediation object in detail, a combination of text annotation and graphic annotation can be selected; in a scene that highlights specific information, color annotation or highlighting can be selected.

[0053] After the annotation is completed, the computer device outputs the annotated image to a display device or storage medium. The output format can be selected according to actual needs, such as common image formats such as JPEG and PNG. At the same time, in order to facilitate relevant personnel to view and analyze the annotation results, the computer device can also provide interactive functions such as zooming in, zooming out, and panning.

[0054] Step S105 is a key step of visually presenting the analysis results. By marking the target mediation room image based on the mediation object and the visual relationship, the computer device enables the key information of the mediation process to be presented in an intuitive and understandable manner.

[0055] Please refer to Figure 2 , is a schematic diagram of the training process of each network component provided in an embodiment of the present application, wherein the debugging process of the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component includes the following steps:

[0056] Step S10: Obtain an image sample and a priori annotations corresponding to the image sample; the priori annotations include mediation object annotations corresponding to the image sample and visual relationship annotations corresponding to the image sample.

[0057] Step S10 provides a solid foundation for subsequent model training by collecting high-quality image samples and corresponding prior annotations. These annotated data not only contain the identity information of the mediation objects, but also cover the visual relationship between the mediation objects, providing rich information for the model to learn complex visual features and relationship patterns.

[0058] Image samples are the basic data for training machine learning models. In remote video mediation scenarios, these image samples come from video surveillance or meeting records in the mediation room, for example. The computer device extracts representative video frames from these video sources as image samples. These video frames should contain rich visual information, such as the facial expressions, gestures, and interactive behaviors of the mediation object.

[0059] Prior annotation is a detailed description of key information in the image sample, including the identity annotation of the mediation object and the visual relationship annotation. Identity annotation of the mediation object, for example, involves identifying and marking the identity of each person in the image, such as "mediator", "plaintiff", "defendant", etc. Visual relationship annotation is more complicated, which involves understanding and describing the interactive behavior between the mediation objects, such as "the mediator is listening to the plaintiff's statement", "the defendant seems nervous", etc. In order to obtain these prior annotations, the computer device may use a manual annotation team or an automated annotation tool. The manual annotation team, for example, is composed of professional annotators who watch the video frames and annotate the key information in the image according to the preset annotation specifications. Automated annotation tools use machine learning technology to assist or replace the manual annotation process to improve the efficiency and accuracy of annotation. For visual relationship annotation, the annotation team will further analyze the interactive behavior between the mediation objects and describe these relationships using natural language or predefined visual relationship tags. For example, in an image showing a mediator communicating with a plaintiff, the annotator may annotate a visual relationship such as "mediator-listening-plaintiff".

[0060] Step S20: extracting features from the image sample based on the feature extraction component to obtain a sample feature vector corresponding to the image sample.

[0061] Through feature extraction, the computer device can extract useful feature information from the original image, such as edges, textures, colors, etc. This information is crucial for the model to understand the image content and conduct subsequent analysis. In the remote video mediation scenario, the feature extraction component can capture key information such as the appearance characteristics, expression changes, and gestures of the mediation object, providing strong support for subsequent mediation object recognition and visual relationship detection.

[0062] The computer device first loads a feature extraction component that has been designed and debugged. This component is, for example, a pre-trained deep learning model, such as a convolutional neural network (CNN), or may be other types of feature extractors.

[0063] The collected image samples are input into the feature extraction component. These image samples should contain rich visual information so that the feature extraction component can extract useful features from them.

[0064] The feature extraction component processes the input image samples and extracts key features in the image through operations such as convolution and pooling. These features may include edges, textures, color histograms, etc.

[0065] The extracted feature information is integrated into a high-dimensional vector representation, namely the sample feature vector. This vector contains all the key feature information of the image sample and can be used for subsequent model training and analysis. For example, suppose in a remote family dispute mediation scenario, the computer device extracts image samples from the mediation room video and uses the feature extraction component to extract features from these samples. The feature extraction component is a convolutional neural network-based model that has been pre-trained on a large-scale image dataset.

[0066] The computer device loads a pre-trained convolutional neural network model as a feature extraction component.

[0067] An image containing the mediator and the parties is captured from the mediation room video as a sample input into the feature extraction component. The convolutional neural network model performs convolution operations on the input image to extract features such as edges and textures in the image. Then, the features are reduced in dimension and selected through pooling operations to reduce the amount of calculation and improve generalization ability. The extracted feature information is integrated into a high-dimensional vector, such as [0.12, 0.34, 0.56, ..., 0.78]. This vector contains all the key feature information of the image sample and can be used for subsequent mediation object recognition and visual relationship detection.

[0068] During the execution of step S20, the selection of feature extraction components is crucial to the success of subsequent steps. Viable feature extraction components include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, etc. When selecting, factors such as model complexity, computational efficiency, and generalization ability are considered. The feature extraction method determines how to extract useful feature information from the image. In convolutional neural networks, feasible feature extraction methods include convolution operations, pooling operations, activation functions, etc. These operations can extract features from the image layer by layer, from low-level edges and textures to high-level objects and scenes.

[0069] The sample feature vector is the output of the feature extraction component, which contains all the key feature information of the image sample. This vector is, for example, a high-dimensional numerical vector, where each element represents the value of a feature. In practical applications, the feature vector may be normalized, dimensionally reduced, etc. to improve the efficiency and accuracy of subsequent steps.

[0070] The performance of the feature extraction component can be improved through training and optimization. Viable optimization methods include back propagation algorithm, gradient descent method, etc. During the training process, a large amount of labeled data is used to guide the model to learn how to extract useful feature information.

[0071] Step S30: Identify the mediation object of the image sample according to the sample feature vector and the mediation object identification component to obtain the identified mediation object.

[0072] In the remote video mediation scenario, accurate identification of the mediation object is the basis for realizing the automated mediation process. The mediation object may include the mediator, the parties, the audience, etc. Their identity, behavior, emotion and other information are of great significance for understanding the mediation process and evaluating the mediation effect. Step S30 can automatically identify the mediation object in the image sample by using the sample feature vector and the mediation object recognition component, and provide accurate data support for subsequent analysis and intervention. The computer device first loads the mediation object recognition component that has been designed and debugged. This component is, for example, a classifier based on deep learning, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), or it may be other types of classification algorithms. The sample feature vector extracted in step S20 is input into the mediation object recognition component. The mediation object recognition component processes the input sample feature vector and identifies the mediation object in the image sample through a classification algorithm. This process may involve complex calculations and reasoning, such as forward propagation and back propagation of a multi-layer neural network. The recognition result is output as the identification of the mediation object. These recognition results include, for example, information such as the identity, location, and quantity of the mediation object, which can be used for subsequent analysis and intervention.

[0073] The computer device loads a pre-trained convolutional neural network classifier as a mediation object recognition component. The sample feature vector extracted from step S20, such as [0.12, 0.34, 0.56, ..., 0.78], is input into the mediation object recognition component. The convolutional neural network classifier processes the input sample feature vector and outputs the recognition result through multi-layer convolution, pooling and forward propagation of the fully connected layer. For example, the output is a probability distribution vector, such as [0.1, 0.8, 0.1], indicating that the probabilities of the mediator, the plaintiff and the defendant in the image sample are 10%, 80% and 10% respectively.

[0074] According to the output probability distribution vector, the computer device selects the category with the highest probability as the recognition result, that is, recognizes that the mediation object in the image sample is the "plaintiff".

[0075] During the execution of step S30, the selection of the mediation object recognition component is crucial to the recognition effect. Feasible components include convolutional neural network (CNN), recurrent neural network (RNN), support vector machine (SVM), etc. When selecting, factors such as model complexity, computational efficiency, and generalization ability are considered. The classification algorithm is the core of the mediation object recognition component. In the convolutional neural network, the classification algorithm includes, for example, multiple layers of convolutional layers, pooling layers, and fully connected layers. The convolutional layer is used to extract image features, the pooling layer is used for dimensionality reduction and feature selection, and the fully connected layer is used for classification decisions. During the training process, the model parameters are optimized by the back propagation algorithm and the gradient descent method to improve the classification accuracy. The sample feature vector is the input data of the mediation object recognition component. In practical applications, the feature vector may be preprocessed, such as normalization, dimensionality reduction, etc., to improve the efficiency and accuracy of the classification algorithm.

[0076] The recognition result is output, for example, in the form of a probability distribution vector, which represents the probability of different categories. In practical applications, the recognition result may be interpreted and evaluated according to specific needs. For example, a threshold may be set to determine the confidence of the recognition result, or indicators such as a confusion matrix and ROC curve may be used to evaluate the performance of the model.

[0077] Step S40: Identify the mediation object visual relationship of the image sample according to the mediation object identification and visual relationship detection component to obtain the identified visual relationship.

[0078] In remote video mediation scenarios, the visual relationship between the mediation objects often contains rich information, such as their interaction patterns, emotional states, behavioral trends, etc. This information is of great significance for a comprehensive understanding of the mediation process, timely detection of potential problems, and effective intervention measures. The computer device loads a visual relationship detection component that has been designed and debugged. This component may be a graph convolutional neural network (GCN), a relation network, etc., or it may be a combination model based on traditional machine learning algorithms.

[0079] The information of the mediation objects identified in step S30 is input into the visual relationship detection component. Such information includes, for example, the identity, location, quantity, etc. of the mediation objects, which is the basis for analyzing the visual relationship of the mediation objects.

[0080] The visual relationship detection component processes the input mediation object information and identifies the visual relationship between them by analyzing the spatial relationship, action, expression and other features between the mediation objects. This process may involve complex calculations and reasoning, such as forward propagation and back propagation of multi-layer neural networks. The recognition results are output as recognized visual relationships. These recognition results can be represented by vectors, where each element represents the strength or probability of a visual relationship. For example, a vector containing multiple elements can represent different visual relationships and their strengths between mediation objects.

[0081] The computer device loads a pre-trained graph convolutional neural network model as a visual relationship detection component. The mediation object information identified in step S30, such as the identity and location of the mediator, the plaintiff, and the defendant, is input into the visual relationship detection component. The graph convolutional neural network model processes the input mediation object information and identifies the visual relationship between them by analyzing the spatial relationship and action characteristics between the mediation objects. For example, the model may recognize that the mediator is actively listening to the plaintiff's statement, while the defendant appears to be nervous. The recognition result is output as a vector, such as [0.8, 0.2, 0.1], indicating that the strength of the "mediator-listening-plaintiff" relationship is 0.8, the strength of the "defendant-nervous" relationship is 0.2, and the strength of other relationships is lower.

[0082] Step S50: Determine a first debugging cost based on an error between identifying a mediation object and annotating the mediation object, determine a second debugging cost based on an error between identifying a visual relationship and annotating the visual relationship, determine a first optimization direction based on the first debugging cost, determine a second optimization direction based on the second debugging cost, fuse the first optimization direction and the second optimization direction to obtain a fused optimization direction, and collaboratively debug a feature extraction component, a mediation object recognition component, and a visual relationship detection component based on the fused optimization direction to optimize the feature extraction component, the mediation object recognition component, and the visual relationship detection component.

[0083] During the training process of the machine learning model, the performance improvement of the model depends on the effective feedback and optimization of the training error. Step S50 provides a clear direction for the optimization of the model by calculating the error between the identified mediation object and the mediation object annotation (the first debugging cost) and the error between the identified visual relationship and the visual relationship annotation (the second debugging cost). These debugging costs reflect the gap between the current performance of the model and the ideal performance, and are an important basis for guiding the adjustment of model parameters. By continuously optimizing these components, the recognition accuracy and robustness of the model in remote video mediation scenarios can be significantly improved.

[0084] The computer device first calculates the error between the identified mediation object and the mediation object labeling to obtain a first debugging cost. This step is implemented, for example, by calculating a loss function (such as a cross entropy loss), which measures the difference between the probability distribution predicted by the model and the true labeling. For example, assuming that the identified mediation object is [mediator, plaintiff, defendant], and the true labeling is [mediator, plaintiff, unrelated person], the first debugging cost can be obtained by calculating the cross entropy between the two probability distributions. Next, the computer device calculates the error between the identified visual relationship and the visual relationship labeling to obtain a second debugging cost. This step also relies on the calculation of the loss function, but at this time the loss function may consider the complexity and diversity of the visual relationship. For example, a multi-label classification loss function can be used to process multiple labels in the visual relationship labeling. According to the first debugging cost and the second debugging cost, the computer device can respectively determine the optimization direction for the feature extraction component, the mediation object recognition component, and the visual relationship detection component. These optimization directions are, for example, expressed as gradient information, indicating how the model parameters should be adjusted to reduce errors. For example, for the mediation object recognition component, the gradient information may indicate that some neuron weights are increased or decreased to more accurately identify the mediation object.

[0085] Since there is a dependency relationship between the feature extraction component, the mediation object recognition component, and the visual relationship detection component, the computer device fuses their optimization directions to obtain a unified fusion optimization direction. This process may involve the allocation of weights and the synthesis of gradients to ensure the coordinated optimization between the components. Finally, the computer device collaboratively debugs the feature extraction component, the mediation object recognition component, and the visual relationship detection component according to the fusion optimization direction. This process involves, for example, updating model parameters and adjusting the learning rate to gradually reduce the debugging cost and improve model performance.

[0086] For example, the computer device calculates the cross entropy loss between the identification of the mediation object and the mediation object annotation as the first debugging cost, and calculates the multi-label classification loss between the identification of the visual relationship and the visual relationship annotation as the second debugging cost. Subsequently, the device calculates the gradient information of each component through the back propagation algorithm and determines the optimization direction. When fusing the optimization direction, the device takes into account the dependencies between the components and assigns different weights to them. Finally, the device updates the model parameters according to the fused optimization direction and adjusts the learning rate to accelerate the convergence process.

[0087] Step S60: Load the image sample into the optimized feature extraction component, extract features from the image sample based on the optimized feature extraction component to obtain a feature vector, and integrate the recognized visual relationship with the feature vector to optimize the sample feature vector.

[0088] In the training process of the machine learning model, the quality of feature extraction directly affects the recognition accuracy and generalization ability of the model. Step S60 optimizes the feature extraction component and integrates the recognition of visual relationships with the feature vector to improve the information content and discrimination of the feature vector. This can not only improve the performance of the mediation object recognition component and the visual relationship detection component, but also help the entire device better capture key information in the mediation process.

[0089] The computer device first loads the feature extraction component optimized by step S50. This component has been iteratively trained and adjusted to more accurately extract features from image samples. The image samples are input into the optimized feature extraction component for feature extraction. This process converts the image samples into high-dimensional feature vectors, which contain key information of the image, such as edges, textures, colors, etc. The visual relationship identified in step S40 is integrated with the feature vector. This step aims to combine the high-level semantic information of visual relationship with the feature vector to enhance the expressive power of the feature vector. The integration method may include encoding the visual relationship as an additional feature dimension, or weighted fusion of the visual relationship with the feature vector. By integrating the visual relationship information, the sample feature vector is optimized. The optimized feature vector not only contains the low-level visual features of the image, but also incorporates the high-level semantic information of the visual relationship between the mediating objects, thereby improving the information content and discrimination of the feature vector.

[0090] Step S70: Jump to the step of identifying the mediation object of the image sample based on the sample feature vector and the mediation object recognition component to perform repeated debugging. When the debugging cutoff requirement is met, the debugged target feature extraction component, target mediation object recognition component and target visual relationship detection component are obtained. The target feature extraction component, target mediation object recognition component and target visual relationship detection component are used to jointly detect the visual relationship of the mediation object of the target mediation room image.

[0091] Step S70 optimizes the feature extraction component, the mediation object recognition component and the visual relationship detection component through continuous iteration, so that they can more accurately extract features from the image sample, recognize mediation objects and visual relationships. This process not only improves the performance of a single component, but also ensures the coordination between the components, thereby improving the recognition accuracy and robustness of the entire device.

[0092] Before starting debugging, the computer device initializes some key debugging parameters, such as learning rate, batch size, number of iterations, etc. These parameters will affect the efficiency and effect of the debugging process. The computer device will jump to step S20, and extract features from the image sample according to the current feature extraction component to obtain a sample feature vector. Then, the mediation object recognition component is used to identify the mediation object in the image sample based on the sample feature vector. Next, the visual relationship detection component is used to further identify the visual relationship between the mediation objects. This process will be repeated, and each iteration will adjust the model parameters according to the error between the recognition result and the labeled data. After each iteration, the computer device will evaluate the performance of the current model, for example, by calculating the loss function value, accuracy, recall rate and other indicators. These indicators will reflect the performance of the model in identifying mediation objects and visual relationships. The computer device will determine whether to stop debugging according to the preset debugging cutoff requirements. These requirements may include the loss function value being lower than a certain threshold, the accuracy being higher than a certain threshold, the number of iterations reaching an upper limit, etc. If these requirements are met, the debugging process ends; otherwise, iterative debugging will continue. When the debugging process is finished, the computer device will obtain the debugged target feature extraction component, target mediation object recognition component and target visual relationship detection component. These components have been fully trained and optimized and can work together to accurately recognize the mediation objects and visual relationships in the target mediation room image.

[0093] For example, in step S70, the computer device first initializes the debugging parameters, such as a learning rate of 0.001, a batch size of 32, and an upper limit of 10,000 iterations. Then, the device begins to repeatedly debug each component. In each iteration, the device extracts the feature vector of the image sample based on the current feature extraction component, and uses the mediation object recognition component and the visual relationship detection component for recognition. Next, the device calculates indicators such as the loss function value and the accuracy rate to evaluate the model performance. If the loss function value is less than 0.1 and the accuracy rate is higher than 90%, the device considers that the debugging cutoff requirements have been met, stops iterating, and saves the debugged components. Finally, the device obtains the debugged target feature extraction component, target mediation object recognition component, and target visual relationship detection component, which can work together to accurately identify the mediation objects and visual relationships in the process of family dispute mediation.

[0094] By continuously iterating and optimizing these components, the computer device is able to build an accurate and robust remote video mediation device. This process not only improves the performance of individual components, but also ensures the coordination between the components, providing strong support for subsequent mediation process analysis and intervention.

[0095] In one embodiment, before obtaining the image sample and the prior annotation corresponding to the image sample in step S10, the method further includes:

[0096] Step S1: performing image segmentation on the image sample to obtain a corresponding image object set;

[0097] Step S2: comparing the image object set with a pre-recorded mediation object library, and determining a plurality of first mediation object labels corresponding to the image sample based on the comparison result; the pre-recorded mediation object library is obtained based on a pre-deployed visual vocabulary;

[0098] Step S3: determining a plurality of mediation object visual relationships corresponding to first mediation object annotations according to the visual vocabulary, and obtaining first visual relationship annotations corresponding to the image sample.

[0099] Image segmentation divides an image into several regions or objects with similar properties. In remote video mediation scenarios, image segmentation helps separate different objects in the mediation room image (such as the mediator, the parties, the background, etc.), providing a basis for subsequent object comparison and visual relationship determination.

[0100] The execution process may be as follows: a computer device loads image samples for image segmentation. These samples are, for example, key frames or screenshots from a mediation room video. A pre-trained image segmentation model is applied to process the image samples. This model is, for example, a segmentation network based on deep learning, such as a fully convolutional network (FCN), U-Net, etc. The image segmentation algorithm divides the image into multiple regions, each corresponding to an image object. These objects are organized into a set, called an image object set. Each object contains information such as its pixel position, size, shape, etc.

[0101] For example, suppose an image sample contains four main objects: the mediator, the plaintiff, the defendant, and the background. The image segmentation algorithm divides the image into four regions corresponding to the four objects. Each object has a unique identifier (such as object 1, object 2, etc.) and a bounding box or mask that describes its location and shape.

[0102] Viable image segmentation models include FCN, U-Net, DeepLab, etc. These models extract image features through convolutional neural networks (CNN) and use deconvolution or upsampling layers to restore feature maps to the original image size, thereby achieving pixel-level classification. In order to improve segmentation accuracy, the segmentation results may also be post-processed, such as morphological operations (erosion, expansion, opening, closing, etc.) or conditional random field (CRF) optimization.

[0103] In step S2, by comparing the image object set with the pre-entered mediation object library, the computer device can automatically identify the mediation objects in the image and assign predefined labels or identifiers to them. Specifically, the computer device can load a pre-built mediation object library. This library contains a series of templates or feature descriptions of mediation objects, which are obtained based on a pre-deployed visual vocabulary. For each object in the image object set, the computer device extracts its features (such as shape, texture, color, etc.) and compares them with the templates in the mediation object library. The comparison process may involve steps such as feature matching and similarity calculation. Based on the comparison results, the computer device assigns a mediation object label to each successfully matched object and generates a number of first mediation object annotations. These annotations contain the identity information (such as mediator, plaintiff, defendant, etc.) and location information (such as bounding box coordinates) of the object.

[0104] For example, assume that the pre-entered mediation object library contains three templates of mediator, plaintiff, and defendant. During the comparison process, object 1 in the image object set successfully matches the mediator template, object 2 successfully matches the plaintiff template, and objects 3 and 4 do not match any template. Therefore, the computer device assigns the labels of "mediator" and "plaintiff" to object 1 and object 2, respectively, and generates the corresponding first mediation object annotation.

[0105] A visual vocabulary is a dictionary of common visual features used to describe objects in an image. When building a mediation object library, a visual vocabulary can be used to extract and represent the features of mediation objects. Feature extraction is a key step in the alignment process. Possible features include local feature descriptors such as SIFT, SURF, and ORB, and global feature descriptors such as HOG and LBP. During the alignment process, the similarity between the image object and the mediation object template is calculated. Possible similarity metrics include Euclidean distance, cosine similarity, Hamming distance, etc.

[0106] In step S3, the visual relationship between the mediation objects (such as spatial position relationship, interactive behavior, etc.) is of great significance for understanding the mediation process. By determining the visual relationship corresponding to the first mediation object annotation, the computer device can further enrich the semantic information of the image sample and provide strong support for subsequent analysis and intervention.

[0107] The computer device loads a pre-deployed visual vocabulary, which contains vocabulary and rules for describing the visual relationship between mediation objects. For each first mediation object annotation, the computer device analyzes the spatial position relationship between its corresponding object and other objects, such as relative position, distance, direction, etc. Based on the visual vocabulary and the spatial relationship analysis results, the computer device determines one or more visual relationship annotations for each first mediation object annotation. These annotations describe information such as the interaction behavior and spatial layout between mediation objects.

[0108] For example, suppose the visual vocabulary contains words such as "facing", "away", and "side by side" that describe spatial relationships. For the mediator and the plaintiff in the first mediation object annotation, the computer device analyzes that the spatial relationship between them is "facing". Therefore, a visual relationship annotation is generated for this pair of annotations: "mediator-facing-plaintiff".

[0109] Spatial relationship analysis is a key step in determining visual relationship annotations. Viable methods include methods based on geometric transformations (such as affine transformations, projective transformations, etc.) and methods based on machine learning (such as support vector machines, random forests, etc.). The construction of the visual vocabulary is supported by domain knowledge and expert experience. It should contain sufficiently rich vocabulary and rules to describe various visual relationships between mediation objects. Visual relationships can be represented in the form of triples (subject-relationship-object), for example. In remote video mediation scenarios, these triplets can provide valuable information for subsequent analysis and intervention.

[0110] In one embodiment, before acquiring the image sample, the method further includes:

[0111] Step S4: determining a candidate visual object in the image object set, where the candidate visual object is a visual object other than the first mediation object annotations;

[0112] Step S5: determining the object adhesion degree of the candidate visual object;

[0113] Step S6: if it is determined that the obtained object adhesion is greater than the set adhesion, the candidate visual object is marked as a second mediation object corresponding to the image sample;

[0114] Step S7: Acquire the mediation object visual relationship corresponding to the second mediation object annotation, and obtain the second visual relationship annotation corresponding to the image sample.

[0115] In the image segmentation step (S1), an image object set has been obtained, which contains the main objects in the image. However, these objects may not completely cover all the important information in the image. The purpose of step S4 is to identify other important visual objects that are not marked as the first mediation object in the image object set, namely candidate visual objects. These objects may contain information that has an important impact on the mediation process, such as specific items in the mediation room, environmental features, etc.

[0116] The computer device first loads the image object set obtained in step S1. For each object in the image object set, the computer device checks whether it has been marked as a first mediation object (such as a mediator, plaintiff, defendant, etc.). If it has not been marked, it is regarded as a candidate visual object. All visual objects that are not marked as first mediation objects are collected to form a candidate visual object set.

[0117] For example, suppose that in an image of a mediation room, the image object set includes three objects: a mediator, a plaintiff, and a defendant, which have been labeled as the first mediation object. In addition, there is an unlabeled folder and a computer in the image. In step S4, the computer device identifies the folder and the computer as candidate visual objects and adds them to the candidate visual object set. When screening the candidate visual objects, the computer device may consider some additional conditions, such as the size, shape, color, etc. of the object, to ensure that only those objects with potential importance are selected.

[0118] In step S5, object adhesion is an indicator for measuring the degree of connection between different objects in the image. In the remote video mediation scenario, the adhesion between objects can reflect their functional relationship, spatial layout or visual continuity (i.e. spatial proximity, functional relevance or visual continuity). By determining the adhesion of candidate visual objects, it is possible to more accurately determine whether they are closely related to the mediation process, thereby deciding whether to mark them as second mediation objects.

[0119] The computer device selects a suitable adhesion measurement method, such as a measurement based on spatial distance, a measurement based on functional relevance, or a measurement based on visual continuity. For each object in the candidate visual object set, the computer device calculates its adhesion with all first mediation objects. This may involve calculating the spatial distance between objects, analyzing their functional relationships, or evaluating their visual continuity. The adhesion values ​​between all candidate visual objects and the first mediation object are organized into a matrix form, called an adhesion matrix. This matrix reflects the closeness between the candidate visual objects and the first mediation object.

[0120] For example, assuming that the candidate visual object set includes two objects, a folder and a computer, and the first mediation object set includes three objects, a mediator, a plaintiff, and a defendant. In step S5, the computer device calculates the adhesion between the folder and the computer and each mediation object. For example, the folder may be close to the mediator's desktop, so it has a high adhesion to the mediator; while the computer may be located in a corner of the mediation room, and has a low adhesion to all mediation objects.

[0121] When choosing a method to measure adhesion, consider the characteristics and needs of the mediation scene. For example, in a mediation room with a tight spatial layout, a measure based on spatial distance may be more effective; whereas in a mediation scene with complex functions, a measure based on functional relevance may be more appropriate. The calculation of adhesion may involve complex mathematical operations and logical reasoning. For example, in the measurement of spatial distance, the Euclidean distance or Manhattan distance may be used to calculate the physical distance between objects; in the measurement of functional relevance, information such as the role and frequency of use of objects in the mediation process may be analyzed.

[0122] In step S6, by comparing the adhesion of the candidate visual objects with the set adhesion threshold, it is possible to determine which candidate visual objects are closely related to the mediation process and decide whether to mark them as the second mediation objects. This step helps to enrich the semantic information of the image sample and improve the accuracy of subsequent mediation object recognition and visual relationship detection.

[0123] The computer device sets a adhesion threshold according to actual needs. This threshold determines which candidate visual objects will be regarded as second mediation objects for annotation. For each candidate visual object, the computer device compares the maximum adhesion (or average adhesion) between it and the first mediation object with the set adhesion threshold. If the adhesion of a candidate visual object is greater than or equal to the set adhesion threshold, the computer device regards it as a second mediation object and adds corresponding annotations to it. These annotations may contain the object's identity information (such as "folder", "computer", etc.) and location information (such as bounding box coordinates).

[0124] For example, continuing the above example, assume that the set adhesion threshold is 0.7 (assuming that the adhesion value is between 0 and 1). In step S6, the computer device compares the adhesion between the folder and the computer with this threshold. Since the folder has a higher adhesion with the mediator (for example, 0.8), it meets the condition of being the second mediation object and is marked accordingly; while the computer has a lower adhesion (for example, 0.4), so it is not marked as the second mediation object.

[0125] The setting of the adhesion threshold is weighed according to the characteristics and needs of the mediation scenario. If the threshold is set too high, some objects closely related to the mediation process may be omitted; if it is set too low, some unimportant objects may be introduced as second mediation objects. When adding the second mediation object annotation, the computer device ensures the accuracy and consistency of the annotation. For example, a unified annotation format and identifier can be used to distinguish different mediation object types.

[0126] In step S7, after the second mediation object is determined, the visual relationship between them and the first mediation object is further analyzed to more fully understand the interaction and connection in the mediation process. By obtaining the mediation object visual relationship corresponding to the second mediation object annotation, more abundant and accurate data support can be provided for subsequent visual relationship detection and analysis.

[0127] The computer device loads a pre-constructed visual vocabulary, which contains vocabulary and rules for describing the visual relationship between mediation objects. For each second mediation object, the computer device analyzes the visual relationship between it and the first mediation object. This may involve the analysis of spatial position relationships (such as relative position, distance, direction, etc.), analysis of interactive behaviors (such as pointing, touching, passing, etc.), and analysis of visual continuity (such as line of sight following, action synchronization, etc.). Based on the visual vocabulary and visual relationship analysis results, the computer device generates corresponding visual relationship annotations for each second mediation object. These annotations describe the specific visual relationship between the second mediation object and the first mediation object.

[0128] For example, continuing the above example, assume that in step S6, the folder is marked as the second mediation object. In step S7, the computer device analyzes the visual relationship between the folder and the mediator. Since the folder is adjacent to the mediator's desktop and contains important files (assuming that this information can be obtained through image content analysis or additional contextual information), it can be determined that there is a "related" or "important" visual relationship between the folder and the mediator. Based on this relationship, the computer device generates a visual relationship annotation for the folder: "folder-related-mediator".

[0129] In one embodiment, the method further comprises:

[0130] Step S80: acquiring an unlabeled image, performing feature extraction on the unlabeled image based on a feature extraction component, and obtaining a feature vector corresponding to the unlabeled image;

[0131] Step S90: identifying the mediation object of the unlabeled image according to the feature vector corresponding to the unlabeled image and the mediation object identification component, and obtaining the identified mediation object corresponding to the unlabeled image;

[0132] Step S100: identifying the visual relationship of the mediation object of the unlabeled image according to the recognition mediation object corresponding to the unlabeled image and the visual relationship detection component, and obtaining the recognition visual relationship corresponding to the unlabeled image;

[0133] Step S110: if the unlabeled image is determined to be a boundary sample based on one or more of the identification mediation object corresponding to the unlabeled image and the identification visual relationship corresponding to the unlabeled image, the unlabeled image is transmitted to the expert device;

[0134] Step S120: when the mediation object annotation and visual relationship annotation corresponding to the unlabeled image confirmed by the expert device are obtained, the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component are debugged again according to the unlabeled image.

[0135] In practical applications, there are often a large amount of unlabeled image data. Although these data are not clearly labeled with mediation objects and visual relationships, they contain rich information and can be used to further optimize machine learning models. Step S80 extracts features from unlabeled images to provide basic data support for subsequent analysis and recognition.

[0136] The computer device randomly or strategically selects one or more unlabeled images from the image data set for processing. Use a trained feature extraction component (such as a convolutional neural network) to extract features from the unlabeled image. This step is similar to step S20. Through multi-layer convolution, pooling and other operations, the image data is converted into a high-dimensional feature vector. The feature extraction component outputs a feature vector corresponding to the unlabeled image, which contains the key information of the image and provides a basis for subsequent analysis and recognition.

[0137] For example, suppose there is an unlabeled image of a mediation room, which contains a mediator, a plaintiff, and a defendant. After the computer device loads this image, it uses a feature extraction component to process it and finally outputs a 1024-dimensional feature vector, such as [0.12, 0.34, ..., 0.78].

[0138] The feature extraction component can be a deep learning model based on a convolutional neural network, which extracts image features through multi-layer convolution and pooling operations.

[0139] In step S90, after acquiring the feature vectors of the unlabeled image, the feature vectors are processed by the mediation object recognition component to recognize the mediation object in the image. This step is the basis for the subsequent analysis of the visual relationship of the mediation object.

[0140] The computer device loads a trained mediation object recognition component, which is, for example, a classifier based on a neural network. The feature vector obtained in step S80 is input into the mediation object recognition component. The mediation object recognition component processes the input feature vector and outputs a recognition result, i.e., the mediation object in the unlabeled image.

[0141] For example, continuing the above example, the computer device inputs the feature vector obtained in step S80 into the mediation object recognition component. The component outputs a recognition result, such as [mediator, plaintiff, defendant], indicating that the three mediation objects, mediator, plaintiff and defendant, are recognized in the image.

[0142] The mediation object recognition component is, for example, a multi-classifier that can output all possible mediation object categories and their probabilities in the image. The recognition result is a list or array containing mediation object categories and corresponding probabilities.

[0143] In step S100, after identifying the mediation objects in the unlabeled image, the visual relationships between these objects are further analyzed to more fully understand the image content. Step S100 achieves this goal through a visual relationship detection component.

[0144] The computer device loads a trained visual relationship detection component, which is, for example, a model based on a graph neural network or a relationship network. The identified mediation objects obtained in step S90 are input into the visual relationship detection component. The visual relationship detection component analyzes the input mediation objects and outputs the visual relationship between them.

[0145] For example, continuing the above example, the computer device inputs the recognition result [mediator, plaintiff, defendant] into the visual relationship detection component. The component outputs a visual relationship, such as [mediator-listening-plaintiff, defendant-nervous], indicating that the mediator is listening to the plaintiff's statement and the defendant appears nervous. The visual relationship is a list or array containing the relationship type and strength between objects, such as [relationship type 1, relationship strength 1, relationship type 2, relationship strength 2, ...].

[0146] During the model training process, we often encounter some samples that are difficult to classify. These samples are called boundary samples. For these samples, the prediction results of the model are often not accurate enough. By transmitting these samples to experts for annotation, the generalization ability and accuracy of the model can be further improved.

[0147] The computer device determines whether the unlabeled image is a boundary sample based on the accuracy of identifying the mediation object and identifying the visual relationship. This can be achieved by calculating the comparison between the prediction result and the preset threshold. If it is determined to be a boundary sample, the unlabeled image and its related information (such as identifying the mediation object and identifying the visual relationship) are transmitted to the expert device.

[0148] For example, suppose that when a computer device processes an unlabeled image, it finds that the accuracy of identifying the mediating object and the accuracy of identifying the visual relationship are both lower than a preset threshold (e.g., 80%). At this time, the device determines that the image is a boundary sample and transmits it to the expert device for further labeling.

[0149] For example, the confidence or accuracy of the prediction result can be calculated to determine whether the sample is a boundary sample. The expert device is, for example, a workstation or software platform dedicated to image annotation, on which the expert can annotate the transmitted image.

[0150] In step S120, by obtaining the expert's annotation information on the boundary samples, the performance of the feature extraction component, the mediation object recognition component and the visual relationship detection component can be further optimized. This step is a key link in the iterative optimization of the model.

[0151] The computer device obtains the mediation object annotation and visual relationship annotation corresponding to the unlabeled image from the expert device. The target feature extraction component, the target mediation object recognition component and the target visual relationship detection component are loaded and debugged. The loaded components are debugged again using the unlabeled image labeled by the expert. This involves, for example, retraining the model and adjusting the parameters. During the debugging process, the performance of the components is continuously evaluated until the preset performance indicators are met.

[0152] In one embodiment, the feature extraction component includes a vectorization network and an embedding network, and the image sample consists of multiple visual objects. Step S20, based on the feature extraction component, performs feature extraction on the image sample to obtain a sample feature vector corresponding to the image sample, including:

[0153] Step S21: vectorizing each visual object based on the vectorization network to obtain an object representation vector corresponding to each visual object;

[0154] Step S22: loading each object representation vector into the embedding network to obtain the object embedding representation vector corresponding to each visual object;

[0155] Step S23: obtaining a sample feature vector corresponding to the image sample based on the object embedding representation vector corresponding to each visual object.

[0156] In the image samples, visual objects (such as mediators, parties, objects, etc.) are key elements for understanding the mediation process. However, the original image data is at the pixel level and cannot be directly used for the calculation of the machine learning model. Therefore, step S21 converts the visual objects into high-dimensional numerical vectors (i.e., object representation vectors) through a vectorization network for subsequent processing and analysis.

[0157] The computer device loads a pre-trained vectorized network model. This model is, for example, a convolutional neural network (CNN) or other type of neural network that is specifically used for vectorized representation of images or objects. For each visual object in the image sample, the computer device first segments it from the original image (if it has not been segmented yet), and then inputs it into the vectorized network. The vectorized network processes the input visual object, extracts the features of the object through multi-layer convolution, pooling and other operations, and converts it into a high-dimensional numerical vector. This vector is the representation vector of the object, which contains key information of the object, such as shape, texture, color, etc. For example, assume that the image sample contains a mediator object. The computer device first segments it from the original image and then inputs it into the vectorized network. After processing the mediator object, the vectorized network outputs a 1024-dimensional object representation vector, such as [0.12, 0.34, ..., 0.78]. This vector represents the position of the mediator object in the feature space.

[0158] For example, a vectorized network is a CNN-based model that extracts local features through the convolution layer, performs feature dimension reduction and selection through the pooling layer, and finally outputs the object representation vector through the fully connected layer. The object representation vector is a high-dimensional numerical vector, and each element represents the strength or importance of the object in a certain feature dimension. This vector can be used for subsequent embedding network processing or as input to a machine learning model.

[0159] In step S22, although the object representation vectors already contain key information of the visual object, they may still be in a high-dimensional and sparse feature space, which is not conducive to subsequent analysis and calculation. Step S22 further converts the object representation vectors into low-dimensional and dense embedded representation vectors through an embedding network to improve the effectiveness of the features and the computational efficiency.

[0160] The computer device loads a pre-trained embedding network model. This model is, for example, an autoencoder, a variational autoencoder, or other types of embedding learning models. The object representation vector generated in step S21 is input into the embedding network. The embedding network processes the input object representation vector and converts it into a low-dimensional and dense embedding representation vector through encoding and decoding processes (or similar transformation processes). This vector is the embedding representation vector of the object, which retains the key information of the object while reducing the dimension and sparsity of the features.

[0161] The image sample, for example, contains multiple visual objects, and the relationships and interactions between them are crucial to understanding the mediation process. Step S23 generates an overall feature vector of the image sample by integrating the embedded representation vectors of each visual object. The computer device collects the embedded representation vectors of all visual objects generated in step S22. These embedded representation vectors are integrated to generate an overall feature vector of the image sample. The integration method may include splicing, weighted summation, attention mechanism, etc. The final output sample feature vector contains the key information of all visual objects in the image sample and their relationships.

[0162] For example, suppose that the image sample contains three visual objects: mediator, plaintiff, and defendant, and their embedded representation vectors are [0.2, 0.4, ..., 0.6] (mediator), [0.1, 0.3, ..., 0.5] (plaintiff), and [0.3, 0.5, ..., 0.7] (defendant). The computer device concatenates and integrates these vectors (or uses other integration methods) to generate an overall sample feature vector, such as [0.2, 0.4, ..., 0.6, 0.1, 0.3, ..., 0.5, 0.3, 0.5, ...,0.7]. This vector represents the overall representation of the image sample in the feature space.

[0163] In one embodiment, step S22, loading each object representation vector into the embedding network to obtain the object embedding representation vector corresponding to each visual object, includes:

[0164] Step S221: based on the spatial position distribution of the visual objects corresponding to each object representation vector, extracting features from each object representation vector according to a first spatial direction to obtain first embedded representation vectors corresponding to each visual object;

[0165] Step S222: based on the spatial position distribution of the visual objects corresponding to each object representation vector, extracting features from each object representation vector according to the second spatial direction to obtain second embedded representation vectors corresponding to each visual object;

[0166] Step S223: Integrate the first embedding representation vector and the second embedding representation vector corresponding to each visual object respectively to obtain the object embedding representation vector corresponding to each visual object respectively.

[0167] In the field of image processing and computer vision, spatial position distribution is one of the key factors in understanding image content. Information such as the position, direction, and relative distance of visual objects in an image is essential for identifying the relationship between objects and understanding the scene structure. Step S221 considers the spatial position distribution of visual objects and extracts features according to a first spatial direction (such as a horizontal direction), thereby capturing the distribution characteristics and mutual relationships of objects in the horizontal space, thereby generating a first embedded representation vector with spatial perception capabilities.

[0168] The computer device first obtains the spatial position distribution information of each visual object in the image sample, which includes, for example, the center point coordinates, bounding box size and direction of the object. According to actual needs and application scenarios, the computer device determines the first spatial direction. When the first spatial direction is a horizontal direction, the device may divide the image into two left and right regions, or sort the objects according to their horizontal positions. Based on the determined spatial direction, the computer device extracts features from each object representation vector. This may involve performing spatial transformations (such as translation, rotation, scaling, etc.) on the object representation vector, applying convolution operations to capture local features, or using models such as graph neural networks to model the spatial relationship between objects. After feature extraction, the computer device generates a corresponding first embedded representation vector for each visual object. These vectors not only contain the feature information of the object itself, but also incorporate the horizontal distribution characteristics of the object in space.

[0169] For example, assume that the image sample contains three visual objects: a mediator, a plaintiff, and a defendant, and their horizontal positions in the image are mediator, plaintiff, and defendant from left to right. In step S221, the computer device first obtains the spatial position information of the three objects, and then extracts features from their object representation vectors in the horizontal direction. For example, the device may apply a horizontal convolution operation to the representation vector of each object to capture the local features of the object in the horizontal space. Finally, the device generates a first embedded representation vector for each object that incorporates the horizontal spatial distribution characteristics.

[0170] During the feature extraction process, the object representation vector may be spatially transformed to match the spatial direction. For example, in the horizontal direction, the object can be moved to a uniform position for comparison by a translation transformation. The convolution layer in the convolutional neural network (CNN) can capture the local features of the image. In step S221, a one-dimensional or two-dimensional convolution operation can be applied to the representation vector of each object to extract spatial features. If the spatial relationship between objects is considered, a graph neural network (GNN) can be used to model it. In a graph neural network, each object can be represented as a node, and the spatial relationship between objects can be represented as an edge.

[0171] Step S222 is similar to step S221. Step S222 aims to capture the distribution characteristics and mutual relationships of objects in the vertical space by considering the spatial position distribution of visual objects and performing feature extraction according to the second spatial direction (such as the vertical direction). This helps to generate a more comprehensive and accurate embedding representation vector, providing stronger support for subsequent analysis and processing.

[0172] According to actual needs and application scenarios, the computer device determines the second spatial direction. When the second spatial direction is a vertical direction, the device may divide the image into two upper and lower areas, or sort the objects according to their vertical positions. Similar to step S221, the computer device extracts features from each object representation vector according to the determined second spatial direction. This may involve spatial transformation of the object representation vector, application of convolution operations, or use of models such as graph neural networks to model the spatial relationship between objects. After feature extraction, the computer device generates a corresponding second embedded representation vector for each visual object. These vectors not only contain the feature information of the object itself, but also incorporate the vertical distribution characteristics of the object in space.

[0173] Similar to step S221, in step S222, spatial transformation and convolution operations may also be performed on the object representation vector to capture features in the vertical space. If the vertical spatial relationship between objects or more complex spatial structural relationships are considered, a graph neural network can be used to model them. In a graph neural network, the vertical distance or other spatial relationship between objects can be represented by the weight of the edge.

[0174] Step S223 is a key step to integrate the first embedding representation vector and the second embedding representation vector generated in steps S221 and S222. By integrating the embedding representation vectors in these two directions, a more comprehensive and accurate object embedding representation vector can be generated. This helps to more accurately identify the mediation object and detect the visual relationship in the subsequent steps.

[0175] The computer device loads the first embedding representation vector and the second embedding representation vector generated in steps S221 and S222. Select a suitable integration method according to actual needs and application scenarios. Such as concatenation, weighted sum, attention mechanism, etc. After the integration operation, the computer device generates a corresponding object embedding representation vector for each visual object. These vectors contain the distribution characteristics of the object in the horizontal and vertical space and the characteristic information of the object itself.

[0176] For example, continuing the above example, assume that in steps S221 and S222, a first embedding representation vector and a second embedding representation vector are generated for the mediator, the plaintiff, and the defendant, respectively. In step S223, the computer device selects a splicing method as the integration method. For the mediator object, if its first embedding representation vector is [0.1, 0.2, ..., 0.9] (length is 10) and its second embedding representation vector is [0.5, 0.6, ..., 0.4] (length is also 10), then the spliced ​​object embedding representation vector is [0.1, 0.2, ..., 0.9, 0.5, 0.6, ..., 0.4] (length is 20). Similarly, corresponding object embedding representation vectors can be generated for the plaintiff and defendant objects.

[0177] Steps S221 to S223 generate a more comprehensive and accurate object embedding representation vector by considering the spatial position distribution of visual objects and extracting and integrating features according to different spatial directions. This process not only captures the feature information of the object itself, but also incorporates the distribution characteristics and mutual relationships of the objects in space. This refined feature extraction method helps to more accurately identify mediation objects and detect visual relationships in subsequent steps, thereby improving the performance and accuracy of the entire device.

[0178] In one embodiment, the mediation object recognition component includes a restoration mapping network and an object mapping network; step S30, identifying the mediation object of the image sample according to the sample feature vector and the mediation object recognition component, and obtaining the identified mediation object, includes:

[0179] Step S31: In the restoration mapping network, based on the spatial position distribution of the visual objects corresponding to each object embedding representation vector, the object embedding representation vector is restored and mapped to obtain the object restoration mapping vector corresponding to each visual object;

[0180] Step S32: In the object mapping network, the mediation object classification mapping is performed on each object restoration mapping vector to obtain the identified mediation object.

[0181] During the feature extraction process, object embedding representation vectors are obtained. Although these vectors contain key information about the visual objects, they may still be in the embedding space, and their representation may not be intuitive or easy to understand. In addition, since the embedding representation vectors may have lost some spatial position information of the original objects, these vectors are restored to a space that is easier to interpret and process before mediating object recognition. Step S31 achieves this goal through a restoration mapping network, which restores these vectors to object restoration mapping vectors based on the object embedding representation vectors and their corresponding visual object spatial position distribution, providing more meaningful input data for subsequent object recognition.

[0182] The computer device loads a pre-trained restoration mapping network model. This model is, for example, a decoder network that can decode the embedded representation vector into a higher-dimensional or easier-to-understand data form. For each object embedding representation vector, the computer device first obtains the spatial position distribution information of the corresponding visual object. This may include the position, size, direction, etc. of the object in the image. The restoration mapping network processes the input object embedding representation vector and gradually restores the original features or higher-dimensional representation of the object through a multi-layer network structure (such as a fully connected layer, a convolutional layer, etc.). During the processing, the network considers the spatial position distribution information of the object to ensure that the restored vector can maintain the original features and spatial relationships of the object. After being processed by the restoration mapping network, each object embedding representation vector is converted into a new vector-the object restoration mapping vector. This vector contains the original features, spatial position information, etc. of the object, providing more comprehensive and accurate data support for subsequent object recognition.

[0183] For example, suppose there is a mediator object whose embedding representation vector is [0.1, 0.2, ..., 0.9] (a vector of length 10), and its corresponding visual object has a position in the image of (x=100, y=200) and a size of (width=50, height=80). In step S31, the computer device inputs this information into the restoration mapping network. The network first encodes and processes this information, and then gradually restores the original features and spatial position information of the mediator through a multi-layer network structure. Finally, the network outputs an object restoration mapping vector, such as [0.05, 0.15, ..., 0.85] (the length may be different from the original embedding representation vector), which not only contains the feature information of the mediator, but also implicitly represents its position and size information in the image.

[0184] The restoration mapping network is, for example, a decoder network based on deep learning, which can decode the embedded representation vector into a higher-dimensional or easier-to-understand data form through a multi-layer network structure. The network structure may include fully connected layers, convolutional layers, deconvolutional layers, etc. In the restoration mapping process, spatial position distribution information can be added to the network as an additional input feature to help the network better restore the original features and spatial relationships of the object. The object restoration mapping vector is a new vector representation that contains the original features of the object, spatial position information, etc., providing more comprehensive and accurate data support for subsequent object recognition.

[0185] After obtaining the object restoration mapping vectors, these vectors are further processed to identify the mediation objects in the image samples. Step S32 achieves this goal through the object mapping network. The object mapping network is a classifier network that can classify and map the input object restoration mapping vectors and output the mediation object category corresponding to each vector. This process is the core step of mediation object recognition, which determines the accuracy and reliability of the final recognition result.

[0186] The computer device loads a pre-trained object mapping network model. This model is, for example, a multi-layer perceptron (MLP), a convolutional neural network (CNN), or another type of classifier network. The object restoration mapping vector generated in step S31 is input into the object mapping network. For each visual object in the image sample, there is a corresponding object restoration mapping vector as input. The object mapping network processes the input object restoration mapping vector, gradually extracts the features of the object through a multi-layer network structure (such as a fully connected layer, a convolutional layer, etc.), and performs classification mapping. During the processing, the network learns how to map different object restoration mapping vectors to corresponding mediation object categories. After being processed by the object mapping network, each object restoration mapping vector is mapped to a mediation object category. The computer device collects these mapping results to obtain recognition results for all mediation objects in the image sample.

[0187] In one embodiment, step S40, identifying the visual relationship of the mediation object of the image sample according to the mediation object identification and visual relationship detection component, to obtain the identified visual relationship, includes:

[0188] Step S41: determining a number of sample mediation objects corresponding to the image sample based on the identified mediation object;

[0189] Step S42: determining a corresponding sample mediation object tuple according to a number of sample mediation objects;

[0190] Step S43: determining the sample binary vector corresponding to the sample mediation object binary in the sample feature vector;

[0191] Step S44: Load the sample binary vector into the visual relationship detection component to perform visual relationship detection, and obtain the identified visual relationship corresponding to the sample mediation object binary.

[0192] During the mediation process, the image sample may contain multiple mediation objects, such as mediators, parties, and observers. The visual relationship between these objects is crucial to understanding the mediation process. The purpose of step S41 is to determine the specific sample mediation object in the image sample based on the mediation object identified in step S30, so as to provide a clear analysis object for subsequent steps.

[0193] The computer device loads the list of mediation objects identified in step S30. This list contains all visual objects in the image sample that are identified as mediation objects and their category labels. Sample mediation objects for analyzing visual relationships are screened out from the list of identified mediation objects. These objects may be screened out according to actual needs, for example, only mediation objects of a specific category (such as mediators and parties) are selected for analysis. The screened sample mediation objects are organized into a set, called a sample mediation object set. This set contains all mediation objects for analyzing visual relationships in the image sample.

[0194] The identified mediation object list is a list of all visual objects in the image sample that are identified as mediation objects and their category labels. The filtering conditions may be based on the category, location, size and other attributes of the mediation object to ensure the accuracy and effectiveness of subsequent analysis. The sample mediation object collection is a collection of all mediation objects for analyzing visual relationships, providing clear analysis objects for subsequent steps.

[0195] The visual relationship between mediation objects is represented, for example, in the form of a tuple, that is, the relationship between two mediation objects. The purpose of step S42 is to generate all possible sample mediation object tuples based on the sample mediation objects determined in step S41, so as to provide basic data for analyzing visual relationships in subsequent steps.

[0196] The computer device loads the sample mediation object set generated in step S41. For each mediation object in the sample mediation object set, it is combined with all other mediation objects to form a tuple. For example, if there are three mediation objects A, B, and C, the generated tuples are (A, B), (A, C), and (B, C). The generated sample mediation object tuples are stored for use in subsequent steps.

[0197] For example, continuing the above example, assuming that the sample mediation object set is {mediator, plaintiff, defendant}, in step S42, the computer device generates the following sample mediation object tuples: (mediator, plaintiff), (mediator, defendant), (plaintiff, defendant), and stores them.

[0198] When generating sample mediation object tuples, a combinatorial algorithm (such as the number of combinations of 2 elements from n different elements) can be used to ensure that all possible tuples are generated. The sample mediation object tuples can be stored in a list, set, or other suitable data structure for efficient access and processing in subsequent steps.

[0199] In the feature extraction process, a sample feature vector is obtained, which contains the feature information of all visual objects in the image sample. The purpose of step S43 is to extract sample binary vectors corresponding to sample mediation object binary vectors from the sample feature vectors, which will be used as input data for the visual relationship detection component.

[0200] The computer device loads the sample feature vector generated in step S20, and locates the feature region corresponding to each sample mediation object binary generated in step S42 in the sample feature vector. This involves, for example, determining the corresponding part of the feature vector based on the position, size, etc. of the sample mediation object in the image. The feature vector segments corresponding to each sample mediation object binary are extracted from the sample feature vector, and these segments are the sample binary vectors.

[0201] For example, assume that the sample feature vector is a long vector that contains the feature information of all visual objects in the image sample. In step S43, the computer device first locates the feature regions corresponding to the sample mediation objects according to their position information in the image. For example, if the positions of the mediator and the plaintiff in the image are (x1, y1) and (x2, y2), respectively, and their sizes are (width1, height1) and (width2, height2), respectively, the computer device can find the feature regions corresponding to the two objects in the sample feature vector. Then, the feature vector fragments corresponding to the (mediator, plaintiff) binary are extracted from these feature regions as the sample binary vector.

[0202] When locating the sample mediation object features in the sample feature vector, the corresponding part of the feature vector may be determined based on the bounding box, mask, or other positioning information of the object. When extracting the sample bigram vector, the overlapping area or contextual information between the objects may be considered to ensure that the extracted vector can accurately reflect the relationship between the objects. The dimension of the sample bigram vector may be different from the dimension of the original sample feature vector, depending on the specific method and requirements for extracting features.

[0203] Step S44 uses the visual relationship detection component to process the sample binary vector extracted in step S43 to identify the visual relationship between the sample mediation object binary. The computer device loads a pre-trained visual relationship detection component model. This model is, for example, a relationship network based on deep learning, a graph neural network, or other types of network structures, which are specifically used to detect the visual relationship between objects in an image. The sample binary vector extracted in step S43 is loaded into the visual relationship detection component as input data. The visual relationship detection component processes the input sample binary vector, and gradually extracts the features between the objects through a multi-layer network structure (such as a convolutional layer, a fully connected layer, a relationship layer, etc.) and performs relationship detection. During the processing, the network learns how to map different sample binary vectors to corresponding visual relationship categories. After being processed by the visual relationship detection component, each sample binary vector is mapped to a visual relationship category. The computer device collects these mapping results to obtain the identified visual relationship corresponding to the sample mediation object binary.

[0204] In one embodiment, the visual relationship detection component includes an intermediate network, a filter network, and a feature compression network; step S44, loading the sample binary vector into the visual relationship detection component for visual relationship detection, and obtaining the identified visual relationship corresponding to the sample mediation object binary, including:

[0205] Step S441: loading the sample binary vector into the intermediate network, processing the sample binary vector based on the intermediate network, and obtaining a corresponding first transition representation vector;

[0206] Step S442: loading the first transition characterization vector into the filter network, and performing a filtering operation on the first transition characterization vector based on the filter network to obtain a corresponding second transition characterization vector;

[0207] Step S443: loading the second transition characterization vector into the feature compression network, and performing a feature compression operation on the second transition characterization vector based on the feature compression network;

[0208] Step S444: Classify the visual relationship of the mediation objects based on the result of the feature compression operation to obtain the recognized visual relationship.

[0209] Although the sample binary vector contains preliminary feature information between the mediated objects, it may still be in the original feature space, and its representation may not be suitable for direct visual relationship detection. The role of the intermediate network is to perform preliminary processing on the sample binary vector and convert it into a representation space that is more suitable for subsequent operations. This process helps to extract more abstract and robust features, providing more meaningful input data for subsequent steps.

[0210] The computer device loads the sample binary vector generated in step S43 into the memory, ready to be input into the intermediate network. The intermediate network processes the input sample binary vector. This processing may include multi-layer convolution operations, pooling operations, activation function applications, etc., to extract key features in the sample binary vector and reduce redundant information. After being processed by the intermediate network, the sample binary vector is converted into a new representation vector - the first transition representation vector. This vector contains more abstract and robust feature information, providing better input data for subsequent steps.

[0211] The intermediate network is, for example, a multi-layer convolutional neural network (CNN), which contains multiple convolutional layers, pooling layers, and activation function layers. The convolutional layer is used to extract local features, the pooling layer is used to reduce the dimension and select key features, and the activation function layer is used to introduce nonlinear factors. The intermediate network extracts the key features in the sample binary vector through convolution and pooling operations, and reduces redundant information through dimensionality reduction. This process helps to improve the computational efficiency and accuracy of subsequent steps. The first transition representation vector is a new vector representation that contains more abstract and robust feature information, providing better input data for subsequent steps.

[0212] Although the first transition characterization vector already contains relatively abstract and robust feature information, it may still contain some noise or redundant information. The function of the filter network is to perform a filtering operation on the first transition characterization vector to further reduce noise and redundant information and improve the purity and effectiveness of the features. This process helps the subsequent steps to more accurately identify the visual relationship between the mediation objects. The computer device loads the first transition characterization vector generated in step S441 into the filter network. The filter network performs a filtering operation on the input first transition characterization vector. This operation may include applying a series of filter kernels, performing convolution operations, applying nonlinear activation functions, and other steps. The design and selection of the filter may be based on prior knowledge or automatic learning. After the filtering operation of the filter network, the first transition characterization vector is converted into a new characterization vector-the second transition characterization vector. This vector contains purer and more effective feature information, providing better input data for subsequent steps.

[0213] The filter network is, for example, a network structure including multiple filter kernels and convolution layers. The design of the filter kernel may be based on prior knowledge (such as Gaussian filter, mean filter, etc.) or automatically learned (such as filters obtained by training convolutional neural networks). The filter network filters the input first transition representation vector through a convolution operation to remove noise and redundant information. The filtering operation may involve steps such as the application of multiple filter kernels and the application of nonlinear activation functions. The second transition representation vector is a new vector representation that contains purer and more effective feature information and provides better input data for subsequent steps. Although the second transition representation vector contains pure and effective feature information, its dimension may still be high, which is not conducive to the calculation and processing of subsequent steps. The function of the feature compression network is to perform feature compression operations on the second transition representation vector to reduce its dimension and retain key feature information. This process helps to reduce the amount of calculation and improve processing speed and accuracy.

[0214] The computer device loads the second transition characterization vector generated in step S442 into the feature compression network. The feature compression network performs a feature compression operation on the input second transition characterization vector. This operation may include applying principal component analysis (PCA), linear discriminant analysis (LDA), autoencoder and other methods to reduce the vector dimension and retain key feature information. After being processed by the feature compression network, the second transition characterization vector is converted into a compressed feature vector. This vector has a lower dimension but retains key feature information, providing better input data for subsequent steps.

[0215] The feature compression network can use a variety of methods to reduce the vector dimension and retain key feature information, such as PCA, LDA, Autoencoder, etc. These methods have their own advantages and disadvantages. In practical applications, the appropriate method should be selected according to the specific situation. The compressed feature vector is a low-dimensional vector representation that retains the key feature information in the second transition representation vector but reduces the dimension, providing better input data for subsequent steps.

[0216] After being processed by the intermediate network, the filter network and the feature compression network, a compressed feature vector is obtained, which contains the key feature information between the mediation objects and has a low dimension. The purpose of step S444 is to classify the visual relationship between the mediation objects based on this feature vector to identify the specific relationship type between them. The computer device loads the compressed feature vector generated in step S443 into the classifier. The classifier performs a classification operation on the input compressed feature vector. This operation may include applying classification algorithms such as support vector machine (SVM), random forest (RandomForest), neural network (NN) to identify the type of visual relationship between the mediation objects. After being processed by the classifier, the device outputs the result of identifying the visual relationship. This result is, for example, a label or category identifier, indicating the specific relationship type between the mediation objects (such as "facing", "staying away", "talking", etc.). The classifier can use a variety of algorithms to implement the classification of the visual relationship of the mediation objects, such as SVM, Random Forest, NN, etc. These algorithms each have advantages and disadvantages and scope of application. In practical applications, the appropriate algorithm is selected according to the specific situation.

[0217] In one embodiment, step S104, identifying the mediation object visual relationship of the target mediation room image according to the identified mediation object and the debugged target visual relationship detection component, to obtain the identified visual relationship, includes:

[0218] Step S1041: determining a number of inference mediation objects corresponding to the target mediation room image based on the identified mediation object;

[0219] Step S1042: determining a target mediation identity among a plurality of inference mediation objects, and constructing an inference mediation object tuple by combining the target mediation identity with each of the other inference mediation objects;

[0220] Step S1043: determining the binary vectors corresponding to each binary of the inference mediation object in the feature vector;

[0221] Step S1044: Load each binary vector into the target visual relationship detection component to perform visual relationship detection, and obtain the identified visual relationship corresponding to each inference mediation object binary.

[0222] Step S1041 further determines the inference mediation objects for analyzing visual relationships in the target mediation room image based on the mediation objects identified in step S30. These inference mediation objects will be the basis for constructing inference mediation object tuples and performing visual relationship detection in subsequent steps.

[0223] The computer device loads the list of mediation objects identified in step S30. This list contains all visual objects identified as mediation objects in the target mediation room image and their category labels. Inference mediation objects for analyzing visual relationships are screened out from the list of identified mediation objects. The screening process may be based on the category, position, size and other attributes of the mediation objects, as well as actual needs. For example, only mediation objects of a specific category (such as mediators and main parties) may be selected as inference mediation objects. The screened inference mediation objects are organized into a set, called an inference mediation object set. This set contains all mediation objects for analyzing visual relationships in the target mediation room image.

[0224] The visual relationship between the mediation objects is, for example, represented in the form of a binary group, that is, the relationship between the two mediation objects. In the case of multiple mediation objects, a target mediation identity is determined, and it is constructed into a binary group with other mediation objects respectively, so that the subsequent steps can detect the visual relationship. A mediation object is selected from the set of inference mediation objects as the target mediation identity. This target mediation identity is, for example, a core figure in the mediation process, such as a mediator. The target mediation identity is constructed into a binary group with each other mediation object in the set of inference mediation objects respectively. For example, if the target mediation identity is a mediator, and the set of inference mediation objects includes a plaintiff and a defendant, the constructed binary groups are (mediator, plaintiff) and (mediator, defendant). The constructed binary groups of inference mediation objects are stored for use in subsequent steps. For example, continuing the above example, it is assumed that the set of inference mediation objects is {mediator, plaintiff, defendant}, and the mediator has been determined to be the target mediation identity. In step S1042, the computer device constructs the following binary groups of inference mediation objects: (mediator, plaintiff) and (mediator, defendant), and stores them.

[0225] In the feature extraction process, a feature vector of the target mediation room image is obtained, which contains the feature information of all visual objects in the image. The purpose of step S1043 is to extract the two-tuple vectors corresponding to each inference mediation object two-tuple from the feature vector, and these vectors will be used as input data for the target visual relationship detection component.

[0226] The computer device loads the feature vector of the target mediation room image generated in step S20. The feature region corresponding to each inference mediation object binary constructed in step S1042 is located in the feature vector. This involves, for example, determining the corresponding part of the feature vector based on information such as the position and size of the inference mediation object in the image. Feature vector segments corresponding to each inference mediation object binary are extracted from the feature vector, and these segments are binary vectors. The extraction process may involve operations such as cropping, splicing or weighting of the feature vector.

[0227] Step S1044 uses the debugged target visual relationship detection component to process the binary vector extracted in step S1043 to identify the visual relationship between the binary objects of the reasoning mediation. The computer device loads the debugged and optimized target visual relationship detection component model. This model is, for example, a relational network, a graph neural network or other type of network structure based on deep learning, which is specifically used to detect the visual relationship between objects in the image. Each binary vector extracted in step S1043 is loaded into the target visual relationship detection component as input data. The target visual relationship detection component processes the input binary vector, and gradually extracts the features between the objects through a multi-layer network structure (such as a convolutional layer, a fully connected layer, a relational layer, etc.) and performs relationship detection. During the processing, the network uses the parameters and rules learned during the debugging process to accurately identify the visual relationship between the binary. After being processed by the target visual relationship detection component, each binary vector is mapped to a visual relationship category. The computer device collects these mapping results to obtain the recognized visual relationship corresponding to the binary of the reasoning mediation object.

[0228] In one embodiment, the method further comprises:

[0229] Step S106: When the target mediation room image contains a preset warning mark, a mediation intervention trigger instruction is sent to the warning device to enable the warning device to respond to the warning.

[0230] During the remote video mediation process, the interaction between the mediator and the parties may be affected by many factors, such as emotional fluctuations, language conflicts, abnormal behavior, etc. If these factors are not handled promptly and effectively, they may lead to the failure of mediation or adverse consequences. Therefore, step S106 can automatically detect potential problems during the mediation process by introducing an early warning mechanism, and trigger an early warning response in time, provide timely help and support for the mediator and the parties, and ensure the smooth progress of the mediation and the fairness of the results. The computer device first detects whether the target mediation room image contains preset early warning annotations. These early warning annotations may be set based on the features of the mediation object's expression, action, language, etc., and are used to identify potential mediation problems. For example, when the mediation object's emotions become excited or conflicting words appear in the language, the device may mark the corresponding early warning information in the image. After detecting the early warning annotation, the computer device further determines whether to trigger an early warning response to intervene. This judgment may be based on a comprehensive consideration of factors such as the severity and duration of the early warning annotation. For example, if the warning label indicates that the mood of the mediation object is only slightly fluctuating and lasts for a short time, the device may consider not to intervene immediately; but if the warning label indicates that there is a serious language conflict or abnormal behavior between the mediation objects, the device may consider that a warning response should be triggered immediately.

[0231] When intervention is determined, the computer device sends a mediation intervention trigger instruction to the early warning device. This instruction contains relevant information of the early warning mark (such as the type of warning, severity, location of occurrence, etc.) so that the early warning device can accurately understand the early warning situation and take corresponding early warning response measures. After receiving the mediation intervention trigger instruction, the early warning device will respond to the early warning according to the information in the instruction. The early warning response measures may include sound or light alarms, display of early warning information, and notification to the mediator. These measures are intended to promptly remind the mediator and the parties to pay attention to potential problems and guide them to take appropriate actions to solve the problems.

[0232] Figure 3 A hardware entity diagram of a computer device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the hardware entity of the computer device 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

Claims

1. A method for implementing remote video mediation based on big data, characterized in that: The method comprises: Acquire a remote mediation room video, and acquire a target mediation room image in the remote mediation room video; Performing feature extraction on the target mediation room image based on the debugged target feature extraction component to obtain a feature vector corresponding to the target mediation room image; Identify the mediation object of the target mediation room image according to the feature vector and the debugged target mediation object identification component to obtain the identified mediation object; According to the identified mediation object and the debugged target visual relationship detection component, the mediation object visual relationship of the target mediation room image is identified to obtain the identified visual relationship; Marking in the target mediation room image according to the identified mediation object and the identified visual relationship; The debugging process of the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component includes: Acquire an image sample and a priori annotations corresponding to the image sample; the priori annotations include mediation object annotations corresponding to the image sample and visual relationship annotations corresponding to the image sample; Perform feature extraction on the image sample based on a feature extraction component to obtain a sample feature vector corresponding to the image sample; Identify the mediation object of the image sample according to the sample feature vector and the mediation object identification component to obtain the identified mediation object; Identifying the mediation object visual relationship of the image sample according to the mediation object identification and visual relationship detection component to obtain the identified visual relationship; Determine a first debugging cost according to an error between the identified mediation object and the mediation object annotation, determine a second debugging cost according to an error between the identified visual relationship and the visual relationship annotation, determine a first optimization direction according to the first debugging cost, determine a second optimization direction according to the second debugging cost, fuse the first optimization direction and the second optimization direction to obtain a fused optimization direction, and collaboratively debug the feature extraction component, the mediation object recognition component, and the visual relationship detection component according to the fused optimization direction to optimize the feature extraction component, the mediation object recognition component, and the visual relationship detection component; Loading the image sample into the optimized feature extraction component, performing feature extraction on the image sample based on the optimized feature extraction component to obtain a feature vector, and integrating the identified visual relationship with the feature vector to optimize the sample feature vector; Jump to the step of identifying the mediation object of the image sample according to the sample feature vector and the mediation object recognition component to perform repeated debugging. When the debugging cutoff requirement is met, the debugged target feature extraction component, target mediation object recognition component and target visual relationship detection component are obtained. The target feature extraction component, target mediation object recognition component and target visual relationship detection component are used to jointly detect the mediation object visual relationship of the target mediation room image.

2. The method according to claim 1, characterized in that Before acquiring the image sample and the prior annotation corresponding to the image sample, the method further includes: Performing image segmentation on the image sample to obtain a corresponding image object set; Comparing the image object set with a pre-recorded mediation object library, and determining a plurality of first mediation object labels corresponding to the image sample based on the comparison result; the pre-recorded mediation object library is obtained based on a pre-deployed visual vocabulary; The mediation object visual relationships corresponding to the plurality of first mediation object annotations are determined according to the visual vocabulary, and the first visual relationship annotations corresponding to the image sample are obtained.

3. The method according to claim 2, characterized in that Before acquiring the image sample, the method further includes: Determining a candidate visual object in the image object set, the candidate visual object being a visual object other than the plurality of first mediation object annotations; determining an object stickiness degree of the candidate visual object; If it is determined that the obtained object adhesion is greater than the set adhesion, marking the candidate visual object as a second mediation object corresponding to the image sample; The mediation object visual relationship corresponding to the second mediation object annotation is obtained, and the second visual relationship annotation corresponding to the image sample is obtained.

4. The method according to claim 1, characterized in that The method further comprises: Acquire an unlabeled image, perform feature extraction on the unlabeled image based on a feature extraction component, and obtain a feature vector corresponding to the unlabeled image; Identify the mediation object of the unlabeled image according to the feature vector corresponding to the unlabeled image and the mediation object identification component, and obtain the identified mediation object corresponding to the unlabeled image; Identifying the visual relationship of the mediation object of the unlabeled image according to the identified mediation object and the visual relationship detection component corresponding to the unlabeled image, and obtaining the identified visual relationship corresponding to the unlabeled image; If the unlabeled image is determined to be a boundary sample based on one or more of the identified mediation object corresponding to the unlabeled image and the identified visual relationship corresponding to the unlabeled image, transmitting the unlabeled image to an expert device; When the mediation object annotation and visual relationship annotation corresponding to the unlabeled image confirmed by the expert device are obtained, the target feature extraction component, the target mediation object recognition component and the target visual relationship detection component are debugged again according to the unlabeled image; The feature extraction component includes a vectorization network and an embedding network. The image sample consists of a plurality of visual objects. The feature extraction component extracts features from the image sample to obtain a sample feature vector corresponding to the image sample, including: Vectorizing each of the visual objects based on a vectorization network to obtain an object representation vector corresponding to each of the visual objects; Loading each of the object representation vectors into the embedding network to obtain an object embedding representation vector corresponding to each of the visual objects; A sample feature vector corresponding to the image sample is obtained based on the object embedding representation vector corresponding to each of the visual objects.

5. The method according to claim 4, characterized in that The step of loading each of the object representation vectors into the embedding network to obtain the object embedding representation vectors corresponding to each of the visual objects includes: Based on the spatial position distribution of the visual objects corresponding to each of the object representation vectors, feature extraction is performed on each of the object representation vectors according to a first spatial direction to obtain first embedded representation vectors corresponding to each of the visual objects; Based on the spatial position distribution of the visual objects corresponding to each of the object representation vectors, feature extraction is performed on each of the object representation vectors according to a second spatial direction to obtain second embedded representation vectors corresponding to each of the visual objects; The first embedding representation vector and the second embedding representation vector corresponding to each of the visual objects are respectively integrated to obtain the object embedding representation vector corresponding to each of the visual objects.

6. The method according to claim 4, characterized in that The mediation object recognition component includes a restoration mapping network and an object mapping network; the mediation object of the image sample is identified based on the sample feature vector and the mediation object recognition component to obtain the identified mediation object, including: In the restoration mapping network, based on the spatial position distribution of the visual objects corresponding to each of the object embedding representation vectors, the object embedding representation vectors are restored and mapped to obtain object restoration mapping vectors corresponding to each of the visual objects; In the object mapping network, each object restoration mapping vector is subjected to mediation object classification mapping to obtain an identified mediation object.

7. The method according to claim 1, characterized in that The step of identifying the visual relationship of the mediation object of the image sample according to the mediation object identification and visual relationship detection component to obtain the identified visual relationship includes: Determining a plurality of sample mediation objects corresponding to the image sample based on the identified mediation object; Determining corresponding sample mediation object tuples according to the plurality of sample mediation objects; Determining a sample binary vector corresponding to the sample mediation object binary in the sample feature vector; The sample binary vector is loaded into a visual relationship detection component to perform visual relationship detection, and the identified visual relationship corresponding to the sample mediation object binary is obtained.

8. The method according to claim 7, characterized in that The visual relationship detection component includes an intermediate network, a filter network and a feature compression network; the step of loading the sample binary vector into the visual relationship detection component for visual relationship detection to obtain the identified visual relationship corresponding to the sample mediation object binary includes: Loading the sample binary vector into the intermediate network, processing the sample binary vector based on the intermediate network, and obtaining a corresponding first transition representation vector; Loading the first transition characterization vector into the filter network, and performing a filtering operation on the first transition characterization vector based on the filter network to obtain a corresponding second transition characterization vector; Loading the second transition representation vector into the feature compression network, and performing a feature compression operation on the second transition representation vector based on the feature compression network; Based on the result of the feature compression operation, the visual relationship classification of the mediated objects is performed to obtain the identified visual relationship.

9. The method according to claim 1, characterized in that: The step of identifying the mediation object visual relationship of the target mediation room image based on the identified mediation object and the debugged target visual relationship detection component to obtain the identified visual relationship includes: Determine a plurality of inference mediation objects corresponding to the target mediation room image based on the identified mediation object; Determine a target mediation identity among the plurality of inference mediation objects, and construct a reasoning mediation object tuple by respectively combining the target mediation identity with each of the remaining reasoning mediation objects; Determine in the feature vector the binary vectors corresponding to each of the binary tuples of the inference mediation object; Loading each of the two-tuple vectors into a target visual relationship detection component to perform visual relationship detection, and obtaining the identified visual relationship corresponding to each of the two-tuple vectors of the inference mediation object; The method further comprises: When the target mediation room image contains a preset warning mark, a mediation intervention trigger instruction is sent to the warning device to enable the warning device to perform a warning response.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

Images