Visual information display method, device, computer equipment and storage medium

By clustering facial images and video sets and using trigger operations to quickly locate target video groups for playback, the problem of low video browsing efficiency in existing technologies is solved, and efficient video frame search and playback are achieved.

CN114153342BActive Publication Date: 2024-11-26GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010832432.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-18
Publication Date
2024-11-26
Estimated Expiration
2040-08-18

AI Technical Summary

Technical Problem

In existing technologies, the method of finding video frames related to a specific person by playing each video stored in the terminal in its entirety is inefficient.

Method used

By clustering multiple facial images and video sets, facial image groups and video groups are generated, and the target video group is quickly located and played using trigger operations.

Benefits of technology

It improves video browsing efficiency, reduces the total playback time of videos, and enhances user experience and navigation flexibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114153342B_ABST
    Figure CN114153342B_ABST
Patent Text Reader

Abstract

The present application relates to a method, device, computer equipment and storage medium for displaying visual information. The method comprises: obtaining multiple face images and video sets; the video sets include video frames to be displayed; clustering the face images and video sets respectively to obtain at least one face image group consisting of face images, and at least one video group consisting of video frames to be displayed; when a first trigger operation is detected, the face images to be displayed are determined from the face image groups through the first trigger operation; when a second trigger operation is detected for the faces in the face images to be displayed, based on the second trigger operation, a target video group is screened out from the video groups; and the target video group is played. The use of this method can improve the browsing efficiency of images or videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device and storage medium for displaying visual information. Background Art

[0002] With the development of communication technology, more and more applications are integrated into terminals, such as video management applications. Based on the video management application, multiple videos stored in the terminal can be displayed to the user, and each stored video can be played to the user from beginning to end. Since the videos stored in the terminal contain face information, the user can search for video frames related to a specific person in the stored videos.

[0003] Currently, the main method is to accelerate the playback of each video stored in the terminal so that the user can find the video frame related to a specific person in the entire video. However, playing each video from beginning to end will result in low efficiency in video browsing. Summary of the invention

[0004] Based on this, it is necessary to provide a visual information display method, device, computer equipment and storage medium that can improve browsing efficiency in response to the above technical problems.

[0005] A method for displaying visual information, the method comprising:

[0006] Acquire multiple face images and a video set; the video set includes video frames to be displayed;

[0007] Clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed;

[0008] When a first trigger operation is detected, determining a face image to be displayed from the face image group through the first trigger operation;

[0009] When a second trigger operation is detected for a face in the face image pointed to by the first trigger operation, based on the second trigger operation, a target video group is screened out from the video groups;

[0010] Play the target video group.

[0011] In one embodiment, the first trigger operation includes a first click operation and a second click operation; and determining the face image to be displayed from the face image group through the first trigger operation includes:

[0012] When a first click operation on the face image group is detected, selecting a first target face image group from the at least one face image group;

[0013] Determining an arrangement order of each face image in the first target face image group;

[0014] Displaying each face image in the first target face image group according to the arrangement order;

[0015] When a second click operation on the displayed facial image is detected, the facial image pointed to by the second click operation is displayed.

[0016] In one embodiment, the facial image group includes facial images belonging to the same person; and determining the arrangement order of the facial images in the first target facial image group includes:

[0017] Determine the faces of the same person contained in each face image in the first target face image group, which are recorded as clustered faces;

[0018] Determining the correlation between other faces in each face image and the clustered faces;

[0019] Determining the relevance of each face image based on the relevance between other faces and the clustered faces;

[0020] The multiple face images in the first target face image group are sorted according to the relevance of each face image.

[0021] In one embodiment, determining the correlation between other faces in each face image and the clustered faces includes:

[0022] Based on the clustered faces, a topological relationship graph between other faces in each face image and the clustered faces is determined; the topological relationship graph includes at least one edge; the edge in the topological relationship graph is connected to two faces in the same face image;

[0023] Determine the frequency of two different faces connected by edges appearing in each face image;

[0024] Based on the frequency number, determine the weight corresponding to each edge in the topological relationship graph;

[0025] The correlation between other faces in each face image and the clustered faces is determined based on the weights and the connection relationship between the faces in the topological relationship graph.

[0026] In one embodiment, the playing of the target video group includes:

[0027] Determine the video to which each to-be-displayed video frame in the target video group belongs, and obtain the play time of each to-be-displayed video frame in the to-be-displayed video;

[0028] Sorting the to-be-displayed video frames belonging to the same video based on the playback time to obtain at least one intermediate sequence result;

[0029] Sorting all intermediate sequence results according to a preset sorting rule to obtain a video sequence;

[0030] The video sequence is played according to the arrangement order of the video frames to be displayed in the video sequence.

[0031] In one embodiment, playing the video sequence according to the arrangement order of the video frames to be displayed in the video sequence includes:

[0032] When a click operation on the currently played video frame to be displayed is detected, determining the position coordinates pointed to by the click operation;

[0033] Determine the location area to which the click operation belongs based on the location coordinates; the location area includes a first area and a second area;

[0034] When the click operation belongs to the first area, based on the arrangement order of the video frames to be displayed in the video sequence, determining and playing a previous video frame adjacent to the currently played video frame to be displayed and located before the currently played video frame to be displayed;

[0035] When the click operation belongs to the second area, based on the arrangement order of the video frames to be displayed in the video sequence, a subsequent video frame adjacent to the currently played video frame to be displayed and located after the currently played video frame to be displayed is determined and played.

[0036] In one embodiment, the method further comprises:

[0037] When a click operation on a face in the to-be-displayed video frame is detected, a second target face image group is screened out from the at least one face image group according to the click operation;

[0038] The second facial image group is displayed.

[0039] A visual information display device, comprising:

[0040] A clustering module, used to obtain multiple face images and video sets; cluster the face images and the video sets respectively to obtain at least one face image group consisting of face images, and at least one video group consisting of video frames to be displayed;

[0041] An image display module, configured to determine a face image to be displayed from the face image group through the first trigger operation when a first trigger operation is detected;

[0042] The video display module is used to filter out a target video group from the video groups based on the second trigger operation when a second trigger operation is detected on the face in the face image pointed to by the first trigger operation; and play the target video group.

[0043] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Acquire multiple face images and a video set; the video set includes video frames to be displayed;

[0045] Clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed;

[0046] When a first trigger operation is detected, determining a face image to be displayed from the face image group through the first trigger operation;

[0047] When a second trigger operation is detected for a face in the face image pointed to by the first trigger operation, based on the second trigger operation, a target video group is screened out from the video groups;

[0048] Play the target video group.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0050] Acquire multiple face images and a video set; the video set includes video frames to be displayed;

[0051] Clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed;

[0052] When a first trigger operation is detected, determining a face image to be displayed from the face image group through the first trigger operation;

[0053] When a second trigger operation is detected for a face in the face image pointed to by the first trigger operation, based on the second trigger operation, a target video group is screened out from the video groups;

[0054] Play the target video group.

[0055] The above-mentioned visual information display method, device, computer equipment and storage medium can cluster the face images and video sets by acquiring multiple face images and video sets to obtain at least one face image group and at least one video group; by determining the user's trigger operation, when a first trigger operation is detected, the face image to be displayed can be determined according to the first trigger operation, and when a second trigger operation of a face in the face image to be displayed is detected, the target video group is correspondingly screened out, and the playback of the face image to be displayed is automatically jumped to the playback of the target video group. Since the user can select the corresponding video group based on the face in the face image, and browse the selected video group, the video frame associated with the specific person can be quickly found. Compared with the traditional method of requiring a complete browsing of multiple videos, the present application can improve the browsing efficiency of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A diagram showing an application environment of a visual information display method in an embodiment;

[0057] Figure 2 is a flow chart of a visual information display method in one embodiment;

[0058] Figure 3 A schematic diagram of grouping facial images in one embodiment;

[0059] Figure 4 A schematic diagram of jumping from image playback to video playback in one embodiment;

[0060] Figure 5 is a schematic diagram of a method for displaying a facial image to be displayed in one embodiment;

[0061] Figure 6 is a schematic diagram of a topological relationship diagram in an embodiment;

[0062] Figure 7 A schematic diagram of jumping from video playback to image playback in one embodiment;

[0063] Figure 8 is a structural block diagram of a visual information display device in one embodiment;

[0064] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It is understood that the terms "first", "second", etc. used in this application can be used in this article to describe various objects, but these objects are not limited by these terms. These terms are only used to distinguish a first object from another object. For example, without departing from the scope of this application, a first triggering operation can be referred to as a second triggering operation, and similarly, a second triggering operation can be referred to as a first triggering operation. Both the first triggering operation and the second triggering operation are triggering operations, but they are not the same triggering operation.

[0067] The visual information display method provided in this application can be applied to Figure 1 In the application environment shown. The terminal 102 obtains multiple face images and video sets, and clusters the multiple face images and video sets to obtain face image groups and video groups. The terminal displays the face images to be displayed in the face image group according to the user's first trigger operation, and determines and plays the corresponding target video group according to the user's second trigger operation. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0068] In one embodiment, Figure 2 As shown, a visual information display method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0069] S202, obtaining multiple face images and video sets.

[0070] The face image refers to an image containing a face, and the same face image may contain faces of multiple people. The video set includes at least one video stored in the terminal. Some video frames in the video contain the face of at least one person. For the convenience of description, the video frames containing faces are referred to as video frames to be displayed.

[0071] Specifically, the terminal device may be provided with an image library and a video library, wherein the image library stores facial images of multiple characters. The video library pre-stores multiple videos. The facial images may be acquired based on an image acquisition application in the terminal, or may be downloaded from the network. Similarly, the videos may be acquired based on a video acquisition application in the terminal, or may be downloaded from the network.

[0072] In one of the embodiments, the terminal performs face detection on each video frame in the video based on a preset face detection algorithm, and estimates the face quality in the video frame based on indicators such as face clarity, size, angle, and occlusion level, and determines the video frame whose face quality is greater than a preset quality threshold as the video frame to be displayed.

[0073] S204 , clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed.

[0074] Each face image group includes face images belonging to the same person. It is easy to understand that if the same face image contains two faces, the face image will appear in two face image groups at the same time. Each video group includes video frames to be displayed belonging to the same person. It is easy to understand that if the same video frame to be displayed contains two faces, the video frame to be displayed will appear in two video groups at the same time.

[0075] Specifically, a trained feature extraction machine learning model is pre-set in the terminal. The feature extraction machine learning model can have the ability to extract facial features through sample learning. The machine learning model can adopt a neural network model, a dual path network model (DPN, Dual Path Network), a support vector machine or a logistic regression model. The terminal inputs all facial images in the image library and all video frames to be displayed in the video library into the feature extraction machine learning model, and the feature extraction machine learning model extracts the facial features in the facial images and the video frames to be displayed, and clusters the facial features based on a preset clustering algorithm to obtain at least one facial image group and at least one video group. Among them, the preset clustering algorithm can be a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. Among them, facial features are data used to reflect the facial features of a person. Facial features can reflect one or more feature information such as the gender of the person, the contour of the face, hairstyle, glasses, nose, mouth, and the distance between various facial organs.

[0076] In one embodiment, the terminal regards each face image in the image library as a class and calculates the distance between the classes. When the distance between two classes is less than a preset distance threshold, the two classes are merged. The distance between classes can be the distance between the face images contained in each of the two classes, and the distance between two face images can be calculated based on the image information of the two face images (the image information is, for example, a multidimensional vector containing information such as facial features and shooting time). The terminal repeats the above steps of calculating the distance between each class and subsequent steps until no new class is generated. In this way, the clustering operation of multiple face images is completed, and at least one face image grouping is obtained.

[0077] In one embodiment, the feature extraction machine learning model may be a general machine learning model with feature extraction capabilities that has been trained. When the general machine learning model is used for extraction in a specific scenario, the effect is not good, so the general machine learning model needs to be further trained and optimized through samples dedicated to the specific scenario. In this embodiment, the terminal can obtain the model structure and model parameters of the general machine learning model, and import the model parameters into the feature extraction machine learning model structure to obtain a feature extraction machine learning model with model parameters. The model parameters of the feature extraction machine learning model participate in the training as the initial parameters for training the feature extraction machine learning model in this embodiment.

[0078] In one embodiment, the feature extraction machine learning model can be a complex network model formed by multiple layers connected to each other. The feature extraction machine learning model may include multiple feature extraction layers, each feature extraction layer has corresponding model parameters, and each layer may have multiple model parameters. The model parameters in each feature extraction layer perform linear or nonlinear changes on the input face to obtain a feature map as the operation result. Each feature extraction layer receives the operation result of the previous layer, and after its own operation, outputs the operation result of this layer to the next layer until the last feature extraction layer completes the linear or nonlinear change operation. According to the output result of the last feature extraction layer, the face features for the current input face image or video frame are obtained. Among them, the model parameters are the various parameters in the model structure, which can reflect the corresponding relationship between the output and input of each layer of the model.

[0079] In one embodiment, the facial features may include facial texture features. The facial texture features may reflect the depth of pixels of facial organs, including nose, ears, eyebrows, cheeks or lips, etc. The facial texture features may include the color value distribution and brightness value distribution of the pixels of the facial image.

[0080] S206: When a first trigger operation is detected, a face image to be displayed is determined from the face image group through the first trigger operation.

[0081] The trigger operation refers to the user's selection operation of the content displayed in the terminal, and the trigger operation includes but is not limited to a click operation, a swipe operation, a long press operation, etc. The specific operation mode of the trigger operation can be set according to the needs. For example, the operation mode of the trigger operation can be set to a click operation in scenario A, and the operation mode of the trigger operation can be set to a swipe operation in scenario B.

[0082] The image library display interface in the terminal has a character display control. When it is determined that the user clicks the character display control, the terminal obtains at least one face image group generated based on the above clustering operation, and Figure 3 The figure shows the corresponding acquired face image grouping. Figure 3 The schematic diagram of face image grouping in one embodiment. The terminal monitors the trigger operation of the user, and when it is determined that the user has triggered the first trigger operation for the face image grouping, based on the first trigger operation, determines the first target face image grouping from at least one face image grouping. For example, when it is determined that the user has clicked on a face image grouping, the face image grouping clicked by the user is determined as the first target face image grouping. The terminal obtains all face images in the first target face image grouping, and displays all face images in the first target face image grouping to the user accordingly. When it is detected that the user has triggered the first trigger operation for the displayed face image again, the terminal obtains the face image pointed to by the first trigger operation, determines the obtained face image as the face image to be displayed, and displays the face image to be displayed to the user. For example, when it is determined that the user has clicked on a face image in the first target face image grouping, the terminal determines the face image clicked by the user as the face image to be displayed, and displays the face image to be displayed to the user in full screen.

[0083] In one embodiment, when it is determined that the user swipes the face image to be displayed to the left, the terminal determines the arrangement order of the face images in the first target face image group, and determines the next face image to be displayed that is adjacent to the face image to be displayed and is located after the face image to be displayed based on the arrangement order, and at the same time, the terminal displays the next face image to be displayed. Similarly, when it is determined that the user swipes the face image to be displayed to the right, the terminal determines the previous face image to be displayed that is located before the face image to be displayed, and displays the previous face image to be displayed.

[0084] S208: When a second trigger operation on a face in the face image to be displayed is detected, a target video group is screened out from the video groups based on the second trigger operation.

[0085] S210, playing the target video group.

[0086] Specifically, the terminal identifies the face area in the face image to be displayed, and when detecting the second trigger operation of the user on the face area, determines the face pointed to by the second trigger operation. For the convenience of description, the face pointed to by the second trigger operation is referred to as the video face below. For example, when it is detected that the user long presses the face in the face image to be displayed, the face long pressed by the user is determined as the video face.

[0087] After acquiring the video face, the terminal determines the target video group containing the video face in each video frame to be displayed according to the facial features of the video face, and then plays each video frame to be displayed in the target video group.

[0088] Since all video groups and face image groups can be viewed as a network, video groups or face image groups can be viewed as nodes in the network, and any two nodes can be connected based on an edge, users can start from any node and browse the contents of the image library and video library along the edge. This not only greatly improves the browsing efficiency of users, but also improves the browsing effect.

[0089] In a real-time example, Figure 4 As shown, when it is determined that the user long presses the face of the video, the terminal displays a control list, which includes a slow motion control, a delay control, and a video control. When it is determined that the user clicks the slow motion control, the terminal plays the target video group in slow motion; when it is determined that the user clicks the delay control, the terminal plays the target video group with a delay; when the video control is clicked, the terminal plays the target video group at a standard playback speed. Figure 4 FIG. 1 is a schematic diagram of jumping from image playback to video playback in an embodiment.

[0090] It is easy to understand that the animated images stored in the terminal can also be clustered based on the above method to obtain animated image groups, so that it is possible to jump from the face image to be displayed to a animated image group, or jump from the video frame to be displayed to a animated image group.

[0091] In the above-mentioned visual information display method, by acquiring multiple face images and video sets, the face images and video sets can be clustered to obtain at least one face image group and at least one video group; by determining the user's trigger operation, when a first trigger operation is detected, the face image to be displayed can be determined according to the first trigger operation, and when a second trigger operation of a face in the face image to be displayed is detected, the target video group is correspondingly screened out, and the playback of the face image to be displayed is automatically jumped to the playback of the target video group. Since the user can select the corresponding video group based on the face in the face image, and browse the selected video group, the video frame associated with the specific person can be quickly found. Compared with the traditional method of requiring a complete browsing of multiple videos, the present application can improve the browsing efficiency of the video.

[0092] In addition, since only a simple trigger operation is required to quickly switch from image playback to video playback, compared to the traditional method of first exiting image playback and then entering the video library for video playback, this application not only improves the jump efficiency, but also increases the jump flexibility, thereby enhancing the user experience.

[0093] In one embodiment, Figure 5 As shown, through the first trigger operation, determining the face image to be displayed from the face image group includes:

[0094] S502: When a first click operation on a facial image group is detected, a first target facial image group is screened out from at least one facial image group based on the first click operation.

[0095] S504: Determine the arrangement order of the facial images in the first target facial image group.

[0096] S506: Display each face image in the first target face image group in order of arrangement.

[0097] S508: When a second click operation on the displayed facial image is detected, the facial image pointed to by the second click operation is displayed.

[0098] Specifically, when the terminal determines the first click operation on the displayed face image group, the first target face image group is determined based on the first click operation. The terminal obtains all face images in the first target face image group, and sorts the face images based on a preset arrangement rule to obtain a face image sequence. The terminal displays the face images in the order of arrangement of each face image in the face image sequence. When the user's second click operation on the displayed face image is determined, the terminal determines the face image pointed to by the second click operation as the face image to be displayed, and at the same time, the terminal displays the face image to be displayed in full screen.

[0099] In this embodiment, the user only needs to perform a simple click operation to view the face image to be displayed in full screen, which can greatly improve the user experience.

[0100] In one embodiment, determining the arrangement order of each facial image in the first target facial image group includes: determining the faces of the same person contained in each facial image in the first target facial image group, recorded as clustered faces; determining the correlation between other faces in each facial image and the clustered faces; determining the correlation of each facial image based on the correlation between other faces and the clustered faces; and sorting the multiple facial images in the first target facial image group according to the correlation of each facial image.

[0101] Among them, other faces refer to faces of other persons located in the face image except the person corresponding to the clustered faces.

[0102] Specifically, the terminal obtains the facial features of each face in each facial image, performs similarity matching on each facial feature, and determines the same face contained in each facial image in the first target facial image group based on the matching result. For the convenience of description, the same face contained in each facial image in the first target facial image group is referred to as a clustered face. The terminal determines the correlation between other faces in each facial image and the clustered face, and linearly superimposes the correlations of different faces in the same facial image to obtain the correlation of the facial image. The terminal sorts the facial images in the first target facial image from high to low according to the correlation of the facial images, so that the facial image with the highest correlation can be sorted to the head of the team, and the facial image with the lowest correlation can be sorted to the end of the team. When the correlations of the facial images in the first target facial image group are the same, the terminal randomly sorts the facial images with the same correlation.

[0103] For example, the first target face image group contains three face images, namely the first face image, the second face image and the third face image. The first face image contains three different faces, namely face 1, face 2 and face 3; the second face image contains four different faces, namely face 1, face 2, face 4 and face 5; the third face image contains three different faces, namely face 1 and face 3. Among them, face 1 is a clustered face. When the correlation between face 2 and face 1 is 2, the correlation between face 3 and face 1 is 2, the correlation between face 4 and face 1 is 1, and the correlation between face 5 and face 1 is 1, the correlation of the first face image is 2+2=4; the correlation of the second face image is 2+1+1=4; and the correlation of the third face image is 2.

[0104] In this embodiment, since the correlation of facial images is obtained by superimposing the correlations of various faces, the correlation of facial images is more accurate, and by giving priority to displaying facial images with high correlation, not only the browsing efficiency can be improved, but also the user experience can be improved.

[0105] In one embodiment, determining the correlation between other faces in each face image and clustered faces includes: determining a topological relationship graph between other faces in each face image and clustered faces based on clustered faces; the topological relationship graph includes at least one edge; the edge in the topological relationship graph is connected to two faces located in the same face image; determining the frequency of two different faces connected by the edge appearing in each face image; based on the frequency, determining the weight corresponding to each edge in the topological relationship graph; determining the correlation between other faces in each face image and clustered faces according to the weight and the connection relationship between the faces in the topological relationship graph.

[0106] Specifically, the terminal obtains a face image in the first target face image group, and determines different faces in the obtained face image. The terminal uses the clustered faces as a reference, and connects different faces in the obtained face image with an edge. In this way, the terminal repeats the above steps of connecting two different faces with an edge until all face images in the first target face image group are traversed. Thus, the operation of generating a topological relationship graph is completed, and the following is obtained: Figure 6 The topology diagram shown. Figure 6 A schematic diagram of a topological relationship diagram in an embodiment.

[0107] Furthermore, the terminal determines the frequency of two different faces connected by the edge appearing in the same face image, and determines the frequency as the weight value of the corresponding edge. For example, in the above example, when the first face image includes face 1, face 2, and face 3; the second face image includes face 1, face 2, face 4, and face 5; and the third face image includes face 1 and face 3, face 1 and face 2 connected by the edge appear in both the first face image and the second face image. At this time, the weight value of the edge connecting face 1 and face 2 is 2.

[0108] Furthermore, the terminal obtains a preset correlation calculation algorithm, and determines the correlation between each face and the clustered face according to the weight, the connection relationship between each face in the topological relationship diagram, and the correlation calculation algorithm. The correlation calculation algorithm may be: the correlation between face K and the clustered face = (the superposition maximum value of the weights of all edges from the clustered face to face K) / (the square of the number of edges from the clustered face to face K). For example, in the above example, when the topological relationship diagram is as follows Figure 5As shown, when it is necessary to calculate the correlation between face 2 and the clustered face (face 1), the terminal determines based on the topological relationship diagram that there is only one connection path between face 1 and face 2, and the path connects face 1 and face 2 by an edge, and the weight value of the edge is 2. At this time, the terminal divides 2 (the weight value of the edge connecting face 1 and face 2) by 1 (the square of the number of edges between face 1 and face 2) to obtain the correlation 2 between face 1 and face 2.

[0109] It is easy to understand that when there are multiple paths between face K and clustered faces, the terminal superimposes the weight values ​​of the edges contained in the multiple paths respectively to obtain multiple superposition results. The terminal determines the target path with the largest superposition result and calculates the number of edges contained in the target path. Then, the correlation between face K and clustered faces is calculated based on the above correlation calculation algorithm.

[0110] In this embodiment, by generating a topological relationship graph and determining the weight value of each edge in the topological relationship graph, the correlation between other faces in each face image and the clustered faces can be determined based on the weight value and the connection relationship between the faces in the topological relationship graph. This allows faces directly related to the clustered faces to be given a higher correlation, and faces related to the face images to be displayed to be given a relatively lower correlation, so that faces with higher correlation can be displayed preferentially in the future, thereby greatly improving the user experience.

[0111] In one embodiment, playing the target video group includes: determining the video to which each frame of the video frame to be displayed in the target video group belongs, and obtaining the playing time of each frame of the video frame to be displayed in the video to which it belongs; sorting the video frames to be displayed in the same video based on the playing time to obtain at least one intermediate sequence result; sorting all the intermediate sequence results according to a preset sorting rule to obtain a video sequence; and playing the video sequence according to the arrangement order of the video frames to be displayed in the video sequence.

[0112] The playback time of the video frame to be displayed refers to the time taken to play the video frame to be displayed at the standard playback speed starting from the first frame in the video.

[0113] Specifically, when the terminal extracts the video frame to be displayed from the video, the terminal determines the video identifier of the video and the play time of the video frame to be played in the video, and embeds the video identifier and the play time in the video frame to be displayed. When the terminal obtains the target video grouping, the terminal determines the video to which each frame of the video frame to be displayed belongs based on the video identifier embedded in the video frame to be displayed, and sorts the video frames to be displayed belonging to the same video based on the play time embedded in the video frame to be displayed, and obtains at least one intermediate sequence result. For example, when the video frame A to be displayed and the video frame B to be displayed belong to video a, and the play time of the video frame A to be displayed is 10 minutes, and the play time of the video frame B to be displayed is 11 minutes; the video frame C to be displayed and the video frame D to be displayed belong to video b, and the play time of the video frame C to be displayed is 2 minutes, and the play time of the video frame B to be displayed is 7 minutes, the terminal sorts the video frame A to be displayed and the video frame B to be displayed based on the play time, and obtains the intermediate sequence result of AB; sorts the video frame C to be displayed and the video frame D to be displayed based on the play time, and obtains the intermediate sequence result of CD.

[0114] Furthermore, the terminal sorts all the generated intermediate sequence results based on a preset sorting rule to obtain a video sequence, and plays the video sequence according to the arrangement order of each to-be-displayed video frame in the video sequence.

[0115] In one embodiment, the terminal can determine the correlation of each to-be-displayed video frame based on the above correlation calculation algorithm, and superimpose the correlations of the to-be-displayed video frames in the same intermediate sequence result to obtain the correlation of the intermediate sequence result. The terminal sorts the intermediate sequence results according to the correlation of the intermediate sequence results to obtain a video sequence.

[0116] In this embodiment, by playing a sequence of video frames containing the same face, the user can quickly browse the target content in the video without having to watch the entire video, thereby greatly improving the efficiency of the user's video browsing. In addition, the video frames to be displayed are played in a corresponding manner based on the arrangement order in the video sequence, so that the user can watch more important video frames first, thereby improving the user experience.

[0117] In one embodiment, according to the arrangement order of each video frame to be displayed in the video sequence, playing the video sequence includes: when a click operation on the currently played video frame to be displayed is detected, determining the position coordinates pointed to by the click operation; determining the position area to which the click operation belongs based on the position coordinates; the position area includes a first area and a second area; when the click operation belongs to the first area, based on the arrangement order of each video frame to be displayed in the video sequence, determining and playing the previous video frame adjacent to the currently played video frame to be displayed and located before the currently played video frame to be displayed; when the click operation belongs to the second area, based on the arrangement order of each video frame to be displayed in the video sequence, determining and playing the next video frame adjacent to the currently played video frame to be displayed and located after the currently played video frame to be displayed.

[0118] The first area refers to an image area located in the left half of the video frame to be displayed, and the second area refers to an image area located in the right half of the video frame to be displayed.

[0119] Specifically, the terminal can take the central axis of the video frame to be displayed as the starting point, determine the image area to the left of the central axis as the first image area, and determine the image area to the right of the central axis as the second image area. When the terminal plays the current video frame to be displayed, the user can click on the currently played video frame to be displayed, and at the same time, the terminal determines the position coordinates pointed to by the user's click operation. When it is determined that the position coordinates belong to the first area, the terminal obtains the arrangement position of the currently played video frame to be displayed in the video sequence, and obtains the previous video frame based on the arrangement position, and then displays the previous video frame. Among them, the previous video frame refers to the previous video frame that is adjacent to the currently played video frame to be displayed and is located before the currently played video frame to be displayed. Similarly, when it is determined that the user clicks on the second image area, the terminal obtains and displays the next video frame.

[0120] In this embodiment, the user only needs to perform a simple click operation to control the playback of each video frame to be displayed in the video frame, which not only improves the playback efficiency but also greatly improves the user experience.

[0121] In one embodiment, the above-mentioned visual information display method also includes: when a click operation on a face in a video frame to be displayed is detected, based on the click operation, a second target face image group is screened out from at least one face image group; and the second face image group is displayed.

[0122] Specifically, when the terminal plays a video sequence, the terminal monitors the user's click operation. When a click operation on a face in the currently played video frame to be displayed is detected, the terminal obtains the facial features of the face pointed to by the click operation, and determines the second target face image group from the face image group based on the facial features. Each face image in the second target face image group contains the face pointed to by the click operation. At the same time, the terminal sorts the face images in the second face image group and displays the face images in the second face image group in the sorted order.

[0123] In one embodiment, when a user clicks on a face in a currently played video frame to be displayed, the terminal displays a control list including a photo control. When it is determined that the user clicks on the photo control, the terminal randomly obtains a face image including the face pointed to by the click operation, and displays the face image as follows: Figure 7 The obtained face image is shown in full screen. Figure 7 FIG. 1 is a schematic diagram of jumping from video playback to image playback in an embodiment.

[0124] In this embodiment, the user only needs to jump from the video frame to the face image group through a simple click operation. Compared with the traditional method of exiting the video playback first and then entering the image library, this method can reduce the jump steps required for jumping from the video to the image, thereby improving the jump efficiency. In addition, since the user can quickly switch between face image groups or video segments corresponding to different faces based on the face, compared with the traditional method of browsing all images in the image library from beginning to end, or watching the entire video in full, this application can further improve the browsing efficiency.

[0125] It should be understood that although Figure 2 , 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0126] In one embodiment, Figure 8As shown, a visual information display device 800 is provided, including: a clustering module 802, an image display module 804 and a video display module 806, wherein:

[0127] The clustering module 802 is used to obtain multiple face images and video sets; cluster the face images and the video sets to be displayed respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed;

[0128] An image display module 804 is used to determine a face image to be displayed from the face image group through the first trigger operation when a first trigger operation is detected;

[0129] The video display module 806 is used to filter out a target video group from the video groups based on the second trigger operation when a second trigger operation is detected for a face in the face image pointed to by the first trigger operation; and play the target video group.

[0130] In one embodiment, the image display module 804 also includes a sorting module 8041, which is used to, when a first click operation on a face image group is detected, filter out a first target face image group from at least one face image group based on the first click operation; determine the arrangement order of each face image in the first target face image group; display each face image in the first target face image group according to the arrangement order; and when a second click operation on the displayed face image is detected, display the face image pointed to by the second click operation.

[0131] In one embodiment, the sorting module 8041 is also used to determine the faces of the same person contained in each face image in the first target face image group, which are recorded as clustered faces; determine the correlation between other faces in each face image and the clustered faces; determine the correlation of each face image based on the correlation between other faces and the clustered faces; and sort the multiple face images in the first target face image group according to the correlation of each face image.

[0132] In one embodiment, the sorting module 8041 is also used to determine a topological relationship diagram between other faces in each face image and the clustered faces based on the clustered faces; the topological relationship diagram contains at least one edge; the edge in the topological relationship diagram is connected to two faces located in the same face image; determine the frequency of appearance of two different faces connected by the edge in each face image; determine the weight corresponding to each edge in the topological relationship diagram based on the frequency; determine the correlation between other faces in each face image and the clustered faces based on the weight and the connection relationship between the faces in the topological relationship diagram.

[0133] In one embodiment, the video display module 806 also includes a video sequence generation module 8061, which is used to determine the video to which each frame of the video to be displayed in the target video group belongs, and obtain the playback time of each frame of the video to be displayed in the video to which it belongs; sort the video frames to be displayed in the same video based on the playback time to obtain at least one intermediate sequence result; sort all the intermediate sequence results according to a preset sorting rule to obtain a video sequence; and play the video sequence according to the arrangement order of the video frames to be displayed in the video sequence.

[0134] In one embodiment, the video sequence generation module 8061 is also used to determine the position coordinates pointed to by the click operation when a click operation on the currently played video frame to be displayed is detected; determine the position area to which the click operation belongs based on the position coordinates; the position area includes a first area and a second area; when the click operation belongs to the first area, based on the arrangement order of the video frames to be displayed in the video sequence, determine and play the previous video frame that is adjacent to the currently played video frame to be displayed and located before the currently played video frame to be displayed; when the click operation belongs to the second area, based on the arrangement order of the video frames to be displayed in the video sequence, determine and play the next video frame that is adjacent to the currently played video frame to be displayed and located after the currently played video frame to be displayed.

[0135] In one embodiment, the visual information display device 800 is also used to, when a click operation on a face in a video frame to be displayed is detected, filter out a second target face image group from at least one face image group according to the click operation; and display the second face image group.

[0136] For the specific definition of the visual information display device, please refer to the definition of the visual information display method above, which will not be repeated here. Each module in the above-mentioned visual information display device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0137] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a visual information display method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0138] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0140] Acquire multiple face images and a video set; the video set includes video frames to be displayed;

[0141] Clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed;

[0142] When a first trigger operation is detected, determining a face image to be displayed from the face image group through the first trigger operation;

[0143] When a second trigger operation of a human face in the face image to be displayed is detected, based on the second trigger operation, a target video group is screened out from the video groups;

[0144] Play the target video group.

[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0146] When a first click operation on a face image group is detected, based on the first click operation, a first target face image group is screened out from at least one face image group;

[0147] Determining an arrangement order of each face image in the first target face image group;

[0148] Displaying each face image in the first target face image group in order of arrangement;

[0149] When a second click operation on the displayed facial image is detected, the facial image pointed to by the second click operation is displayed.

[0150] In one embodiment, the facial image group includes facial images belonging to the same person; when the processor executes the computer program, the following steps are also implemented:

[0151] Determine the faces of the same person contained in each face image in the first target face image group, and record them as clustered faces;

[0152] Determine the correlation between other faces in each face image and the clustered faces;

[0153] Determining the relevance of each face image based on the relevance between other faces and the clustered faces;

[0154] The multiple face images in the first target face image group are sorted according to the relevance of each face image.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] Based on the clustered faces, a topological relationship graph between other faces in each face image and the clustered faces is determined; the topological relationship graph includes at least one edge; the edge in the topological relationship graph is connected to two faces in the same face image;

[0157] Determine the frequency of two different faces connected by edges appearing in each face image;

[0158] Based on the frequency, determine the weight corresponding to each edge in the topological relationship graph;

[0159] According to the weights and the connection relationship between the faces in the topological relationship diagram, the correlation between other faces in each face image and the clustered faces is determined.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] Determine the video to which each to-be-displayed video frame in the target video group belongs, and obtain the play time of each to-be-displayed video frame in the to-be-displayed video;

[0162] Sorting the to-be-displayed video frames belonging to the same video based on the playback time to obtain at least one intermediate sequence result;

[0163] Sorting all intermediate sequence results according to a preset sorting rule to obtain a video sequence;

[0164] The video sequence is played according to the arrangement order of the video frames to be displayed in the video sequence.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] When a click operation on the currently playing video frame to be displayed is detected, the position coordinates pointed to by the click operation are determined;

[0167] Determine the location area to which the click operation belongs based on the location coordinates; the location area includes a first area and a second area;

[0168] When the click operation belongs to the first area, based on the arrangement order of the video frames to be displayed in the video sequence, a previous video frame adjacent to the currently played video frame to be displayed and located before the currently played video frame to be displayed is determined and played;

[0169] When the click operation belongs to the second area, based on the arrangement order of the video frames to be displayed in the video sequence, a subsequent video frame adjacent to the currently played video frame to be displayed and located after the currently played video frame to be displayed is determined and played.

[0170] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0171] When a click operation on a face in the video frame to be displayed is detected, a second target face image group is screened out from at least one face image group according to the click operation;

[0172] Display the second face image grouping.

[0173] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0174] Acquire multiple face images and a video set; the video set includes video frames to be displayed;

[0175] Clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed;

[0176] When a first trigger operation is detected, determining a face image to be displayed from the face image group through the first trigger operation;

[0177] When a second trigger operation of a human face in the face image to be displayed is detected, based on the second trigger operation, a target video group is screened out from the video groups;

[0178] Play the target video group.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0180] When a first click operation on a face image group is detected, based on the first click operation, a first target face image group is screened out from at least one face image group;

[0181] Determining an arrangement order of each face image in the first target face image group;

[0182] Displaying each face image in the first target face image group in order of arrangement;

[0183] When a second click operation on the displayed facial image is detected, the facial image pointed to by the second click operation is displayed.

[0184] In one embodiment, the facial image group includes facial images belonging to the same person; when the computer program is executed by the processor, the following steps are also implemented:

[0185] Determine the faces of the same person contained in each face image in the first target face image group, and record them as clustered faces;

[0186] Determine the correlation between other faces in each face image and the clustered faces;

[0187] Determining the relevance of each face image based on the relevance between other faces and the clustered faces;

[0188] The multiple face images in the first target face image group are sorted according to the relevance of each face image.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0190] Based on the clustered faces, a topological relationship graph between other faces in each face image and the clustered faces is determined; the topological relationship graph includes at least one edge; the edge in the topological relationship graph is connected to two faces in the same face image;

[0191] Determine the frequency of two different faces connected by edges appearing in each face image;

[0192] Based on the frequency, determine the weight corresponding to each edge in the topological relationship graph;

[0193] The correlation between other faces in each face image and the clustered faces is determined based on the weights and the connection relationship between the faces in the topological relationship graph.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0195] Determine the video to which each to-be-displayed video frame in the target video group belongs, and obtain the play time of each to-be-displayed video frame in the to-be-displayed video;

[0196] Sorting the to-be-displayed video frames belonging to the same video based on the playback time to obtain at least one intermediate sequence result;

[0197] Sorting all intermediate sequence results according to a preset sorting rule to obtain a video sequence;

[0198] The video sequence is played according to the arrangement order of the video frames to be displayed in the video sequence.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] When a click operation on the currently playing video frame to be displayed is detected, the position coordinates pointed to by the click operation are determined;

[0201] Determine the location area to which the click operation belongs based on the location coordinates; the location area includes a first area and a second area;

[0202] When the click operation belongs to the first area, based on the arrangement order of the video frames to be displayed in the video sequence, a previous video frame adjacent to the currently played video frame to be displayed and located before the currently played video frame to be displayed is determined and played;

[0203] When the click operation belongs to the second area, based on the arrangement order of the video frames to be displayed in the video sequence, a subsequent video frame adjacent to the currently played video frame to be displayed and located after the currently played video frame to be displayed is determined and played.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0205] When a click operation on a face in the video frame to be displayed is detected, a second target face image group is screened out from at least one face image group according to the click operation;

[0206] Display the second face image grouping.

[0207] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0208] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0209] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for displaying visual information, It is characterized in that The method comprises: Acquire multiple face images and a video set; the video set includes video frames to be displayed; Clustering the face images and the video set respectively to obtain at least one face image group consisting of face images and at least one video group consisting of video frames to be displayed; each face image group includes face images belonging to the same person; each video group includes video frames to be displayed belonging to the same person; When a first trigger operation is detected, determining a face image to be displayed from the face image group through the first trigger operation; When a second trigger operation on a human face in the face image to be displayed is detected, based on the second trigger operation, a target video group is screened out from the video groups; Play the target video group.

2. The method according to claim 1, It is characterized in that The first trigger operation includes a first click operation and a second click operation; and determining the face image to be displayed from the face image group through the first trigger operation includes: When a first click operation on the face image group is detected, based on the first click operation, a first target face image group is screened out from the at least one face image group; Determining an arrangement order of each face image in the first target face image group; Displaying each face image in the first target face image group according to the arrangement order; When a second click operation on the displayed facial image is detected, the facial image pointed to by the second click operation is displayed.

3. The method according to claim 2, It is characterized in that The facial image group includes facial images belonging to the same person; and determining the arrangement order of the facial images in the first target facial image group includes: Determine the faces of the same person contained in each face image in the first target face image group, which are recorded as clustered faces; Determining the correlation between other faces in each face image and the clustered faces; Determining the relevance of each face image based on the relevance between other faces and the clustered faces; The multiple face images in the first target face image group are sorted according to the relevance of each face image.

4. The method according to claim 3, It is characterized in that Determining the correlation between other faces in each face image and the clustered faces includes: Based on the clustered faces, a topological relationship graph between other faces in each face image and the clustered faces is determined; the topological relationship graph includes at least one edge; the edge in the topological relationship graph is connected to two faces in the same face image; Determine the frequency of two different faces connected by edges appearing in each face image; Based on the frequency number, determine the weight corresponding to each edge in the topological relationship graph; The correlation between other faces in each face image and the clustered faces is determined based on the weights and the connection relationship between the faces in the topological relationship graph.

5. The method according to claim 1, It is characterized in that The playing of the target video group includes: Determine the video to which each to-be-displayed video frame in the target video group belongs, and obtain the play time of each to-be-displayed video frame in the to-be-displayed video; Sorting the to-be-displayed video frames belonging to the same video based on the playback time to obtain at least one intermediate sequence result; Sorting all intermediate sequence results according to a preset sorting rule to obtain a video sequence; The video sequence is played according to the arrangement order of the video frames to be displayed in the video sequence.

6. The method according to claim 5, It is characterized in that Playing the video sequence according to the arrangement order of the video frames to be displayed in the video sequence includes: When a click operation on the currently played video frame to be displayed is detected, determining the position coordinates pointed to by the click operation; Determine the location area to which the click operation belongs based on the location coordinates; the location area includes a first area and a second area; When the click operation belongs to the first area, based on the arrangement order of the video frames to be displayed in the video sequence, determining and playing a previous video frame adjacent to the currently played video frame to be displayed and located before the currently played video frame to be displayed; When the click operation belongs to the second area, based on the arrangement order of the video frames to be displayed in the video sequence, a subsequent video frame adjacent to the currently played video frame to be displayed and located after the currently played video frame to be displayed is determined and played.

7. The method according to claim 1, It is characterized in that The method further comprises: When a click operation on a face in the to-be-displayed video frame is detected, a second target face image group is screened out from the at least one face image group according to the click operation; The second target face image group is displayed.

8. A visual information display device, It is characterized in that The device comprises: A clustering module, used to obtain multiple face images and video sets; cluster the face images and the video sets respectively to obtain at least one face image group consisting of face images, and at least one video group consisting of video frames to be displayed; each face image group includes face images belonging to the same person; each video group includes video frames to be displayed belonging to the same person; An image display module, configured to determine a face image to be displayed from the face image group through the first trigger operation when a first trigger operation is detected; The video display module is used to filter out a target video group from the video groups based on the second trigger operation when a second trigger operation is detected on the face in the face image pointed to by the first trigger operation; and play the target video group.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.