Gait emotion joint recognition method based on heterogeneous space neural architecture search

Abstract

The invention provides a gait and emotion joint recognition method based on heterogeneous space neural architecture search. The method comprises the following steps: extracting three-dimensional human body posture joint points in different emotion video data; carrying out space conversion, respectively converting into a rectangular coordinate system and a spherical coordinate system, and carrying out position information space mapping and angle information space mapping on articulation points in different coordinate systems based on a similarity minimization strategy; constructing a neural architecture search space and a joint search strategy under different spaces, respectively searching to obtain an optimal backbone network, and respectively training the optimal backbone network; utilizing the trained optimal backbone network to extract human body time sequence gait emotion features in different spaces, and inputting the human body time sequence gait emotion features into a Softmax classifier to obtain scores of different emotions; and performing joint scoring to complete gait emotion joint recognition based on heterogeneous space neural architecture search.

Description

Technical Field

[0001] The present invention relates to a gait emotion joint recognition method, in particular to a gait emotion joint recognition method based on heterogeneous space neural architecture search. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Gait emotion recognition is a research field based on computer vision and machine learning technologies, which aims to infer an individual's emotional state by analyzing the way he walks. Studies have shown that emotional state can affect a person's biological characteristics, such as posture, stride, walking speed, and posture changes. This provides a theoretical basis for gait emotion recognition, as emotional state may manifest unique characteristics in walking. With the continuous advancement of the field of artificial intelligence, deep learning related technologies, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), have been widely used in gait emotion recognition. These deep learning methods can automatically extract and learn features, improving the accuracy and robustness of emotion recognition. Gait emotion recognition has a wide range of application prospects, including health monitoring, autonomous driving, virtual reality, emotional intelligence auxiliary tools, etc. The needs of different fields have driven the continuous development and application of gait emotion recognition technology.

[0004] Existing gait emotion recognition based on deep learning mainly extracts the gait information features of the real-time position of the human body through a manually designed deep learning model, and completes gait emotion recognition based on this feature. At present, the acquisition of human gait emotion features is mainly completed by extracting the features of human joints in the rectangular coordinate system through a deep learning model, thereby ignoring the importance of the angle information between human joints to gait emotion features. In addition, the manually designed deep learning model is very limited by the creativity and experience of the designer, and cannot give full play to the potential advantages in the model structure. And when faced with new tasks or data, the manually designed model may need to be redesigned or adjusted, which makes the manually designed model have certain limitations in gait emotion recognition.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a gait emotion joint recognition method based on heterogeneous space neural architecture search in view of the shortcomings of the prior art.

[0007] In order to solve the above technical problems, the present invention discloses a gait emotion joint recognition method based on heterogeneous space neural architecture search, comprising the following steps:

[0008] Step 1, extracting 3D human body posture joint points in different emotion video data;

[0009] Step 2, spatially transform the extracted 3D human body posture joint points into rectangular coordinate system and spherical coordinate system respectively, and perform position information space mapping and angle information space mapping on the joint points in different coordinate systems based on similarity minimization strategy;

[0010] Step 3, constructing neural architecture search spaces and joint search strategies in different spaces, using the joint search strategy, respectively searching for optimal backbone networks in the neural architecture search spaces in different spaces, for extracting spatiotemporal emotion features in the position information space and the angle information space, and respectively training the above optimal backbone networks;

[0011] Step 4: Use the trained optimal backbone network to extract the emotional features of human temporal gait in different spaces and input them into the Softmax classifier to obtain the scores of different emotions;

[0012] Step 5: The different emotion scores in different input spaces are jointly scored through the joint scoring module to complete the joint recognition of gait emotions based on heterogeneous space neural architecture search.

[0013] Furthermore, the three-dimensional human body posture joints described in step 1 include: root joints, spine joints, neck joints, head joints, left shoulder joints, left elbow joints, left hand joints, right shoulder joints, right elbow joints, right hand joints, left hip joints, left knee joints, left foot joints, right hip joints, right knee joints and right foot joints, etc.

[0014] Furthermore, the space conversion described in step 2 specifically includes:

[0015] Convert the three-dimensional human body posture joint points in the rectangular coordinate system extracted in 1 to the spherical coordinate system:

[0016]

[0017]

[0018]

[0019] Among them, any point in the rectangular coordinate system is (x, y, z), and the corresponding point in the transformed spherical coordinate system is

[0020] Furthermore, obtaining the optimal backbone network described in step 3 specifically includes the following steps:

[0021] Step 3-1, constructing a neural architecture search space in different spaces, wherein the different spaces are: a position information space and an angle information space, and the neural architecture search space includes a submodule for constructing a backbone network;

[0022] Step 3-2, construct a joint space search strategy, which is specifically expressed as follows:

[0023]

[0024] Among them, G represents the joint space search strategy, maximize represents maximization, ACC represents the joint gait emotion recognition accuracy, and t c represents the feature extraction speed of the backbone network constructed by each submodule in the neural architecture search space under the position information space, T c represents the predefined feature extraction speed, t s represents the feature extraction speed of the backbone network constructed by each submodule in the neural architecture search space under this angle information space, T s represents the predefined feature extraction speed, w1 and w2 are weight factors, as follows:

[0025]

[0026]

[0027] Among them, α1, β1, α2, β2 are constants;

[0028] Step 3-3, according to the joint search strategy constructed in step 3-2, search for the optimal combination of each sub-module in the neural architecture search space under the position information space and the angle information space as the optimal backbone network.

[0029] Furthermore, the sub-modules for constructing the backbone network described in step 3-1 specifically include: a 3*3 convolution sub-module, a 5*5 convolution sub-module, a 3*3 maximum pooling sub-module, a 3*3 average pooling sub-module, an STGCN sub-module, a channel attention sub-module, and a spatial attention sub-module.

[0030] Furthermore, the step 3-3 of searching for the optimal combination of each submodule in the neural architecture search space under the position information space and the angle information space specifically includes:

[0031] Step 3-3-1, using a controller to search for the initial backbone network in the position information space and the angle information space respectively, wherein the backbone network is a random combination of submodules in the neural architecture search space;

[0032] Step 3-3-2, using the gait emotion dataset to train and evaluate the above initial backbone network, and obtain the joint gait emotion recognition accuracy and the feature extraction speed of the initial backbone network in each space;

[0033] Step 3-3-3, performing calculations according to the joint space search strategy described in step 3-2, and feeding the results back to the controller;

[0034] Step 3-3-4, the controller re-optimizes the search for a new backbone network in the position information space and the angle information space, that is, re-randomly combines the submodules in the neural architecture search space to obtain a new backbone network;

[0035] Step 3-3-5, using the new backbone network as the initial backbone network, repeatedly iteratively execute steps 3-3-2 to 3-3-4 until the preset iteration rounds, and obtain the optimal backbone networks in the position information space and the angle information space respectively.

[0036] Furthermore, the training of the optimal backbone network described in step 3 specifically includes:

[0037] The optimal backbone network is trained using a gait emotion dataset containing emotion labels.

[0038] Furthermore, the scores of different emotions obtained in step 4 specifically include:

[0039] Step 4-1, using the optimal backbone network trained in step 3, extract the sentiment features in the position information space and the angle information space respectively;

[0040] In step 4-2, the Softmax classifier is used to perform sentiment classification and scoring on the sentiment features in different spaces extracted in step 4-1 to obtain scores of different emotions.

[0041] Furthermore, the joint scoring described in step 5 specifically includes:

[0042] The scores of different emotions obtained in step 4 are jointly scored through a joint scoring module to obtain joint scores of different emotions based on different spaces, and the emotion with the highest score is determined as the final result of the emotion recognition.

[0043] Furthermore, the joint scoring module described in step 5 is as follows:

[0044] F i =S i +C i

[0045] Among them, i represents the emotion category, S irepresents the scores of different emotions in the angle information space, C i represents the scores of different emotions in the location information space, F i Represents the joint score output by the joint scoring module.

[0046] Beneficial effects:

[0047] The present invention makes full use of the position information and angle information of three-dimensional human body joints in space, avoids the shortcomings of manually designed deep learning models, fully explores the potential of the model structure, and effectively improves the accuracy and generalization of the gait emotion recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0049] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0050] Figure 2 This is a flowchart of the joint search of backbone networks in different spaces of the present invention.

[0051] Figure 3 This is a schematic diagram of the result of gait skeleton recognition output corresponding to the emotion of frustration. DETAILED DESCRIPTION

[0052] The present invention uses the original three-dimensional posture joint data to map to the three-dimensional posture joint representation in different spaces, and uses the joint search strategy to complete the backbone network search in different spaces, automatically constructs the gait emotion feature extraction network, and completes the gait emotion recognition based on this. The technical solution is as follows:

[0053] A gait emotion joint recognition method based on heterogeneous space neural architecture search specifically includes the following steps:

[0054] Step S1: extracting three-dimensional human body posture joint points in different emotion video data;

[0055] Step S2: performing spatial transformation on the extracted 3D human body posture joint points, respectively transforming them into an angular coordinate system and a spherical coordinate system, and mapping the position information space and the angle information space of the joint points in different coordinate systems based on a similarity minimization strategy;

[0056] Step S3: constructing neural architecture search spaces and joint search strategies in different spaces, using the joint search strategy to jointly search for the backbone network for extracting spatiotemporal emotion features in the position information space and the angle information space, and completing the training based on the gait emotion dataset;

[0057] Step S4: using the backbone network to extract the emotional features of human temporal gait in different spaces, and outputting the scores of different emotions through the Softmax classifier;

[0058] Step S5: The different emotion scores in different input spaces are jointly scored by a joint scoring module to complete gait emotion recognition.

[0059] Preferably, the different emotion video data in step S1 include emotions including happiness, frustration and neutrality.

[0060] Preferably, the three-dimensional human posture joint points extracted in step S1 include root joint points, spine joint points, neck joint points, head joint points, left shoulder joint points, left elbow joint points, left hand joint points, right shoulder joint points, right elbow joint points, right hand joint points, left hip joint points, left knee joint points, left foot joint points, right hip joint points, right knee joint points, right foot joint points, etc.

[0061] Preferably, the space conversion in step S2 includes conversion to position information space and angle information space. The three-dimensional human posture joint points extracted based on the emotional video belong to the rectangular coordinate system, and the three-dimensional human posture joint points in the spherical coordinate system are converted by the conversion formula between the rectangular coordinate system and the spherical coordinate system. In order to avoid the intersection of information spaces, the position information space mapping and the angle information space mapping are completed based on the similarity minimization strategy.

[0062] Preferably, in step S3, constructing a neural architecture search space and a joint search strategy in different spaces, and jointly searching for a backbone network for extracting features in different spaces, specifically includes the following steps:

[0063] Construct a search space including 3*3 convolution, 5*5 convolution, 3*3 maximum pooling, 3*3 average pooling, STGCN module, channel attention module, and spatial attention module;

[0064] The joint space search strategy is constructed, which mainly consists of the joint gait emotion recognition accuracy, the backbone network feature extraction speed in the position information space, and the backbone network feature extraction speed in the angle information space. It can be specifically expressed as:

[0065]

[0066] Where ACC represents the joint gait emotion recognition accuracy, t c represents the backbone network feature extraction speed in the location information space, T c represents the predefined feature extraction speed, t s represents the backbone network feature extraction speed in this angle information space, T srepresents the predefined feature extraction speed, w1 and w1 are weight factors, which can be defined as:

[0067]

[0068]

[0069] Among them, α1, β1, α2, β2 are constants.

[0070] The optimal backbone network in the position information space and angle information space is searched according to the joint search strategy; the entire search process is as follows Figure 2 As shown, it can be specifically described as follows: the controller first searches for the backbone networks in the position information space and the angle information space in the search space, and then uses the gait emotion dataset to train and evaluate the model to obtain the joint gait emotion recognition accuracy and the feature extraction speed of the backbone network in each information space, and then feeds back to the controller according to the search strategy, prompting the controller to re-optimize the search for the backbone networks in the position information space and the angle information space, and performs multiple iterations to obtain the optimal backbone network for extracting features in different spaces.

[0071] The backbone network training is completed using a gait emotion dataset containing happy, depressed, and neutral emotion labels.

[0072] Preferably, the step S4 uses the backbone network to extract the emotional features of human temporal gait in different spaces, and outputs the scores of different emotions through the Softmax classifier. Specifically, it refers to using the trained backbone network to extract the emotional features of the three-dimensional human posture joints in the position information space and the angle information space, and then using the Softmax classifier to perform emotional classification and scoring on the emotional features in different spaces.

[0073] Preferably, the step S5 performs joint scoring on different emotion scores in different spaces input by the joint scoring module, which means that the different emotion scores in different spaces output by Softmax in step S4 are jointly scored, and then the joint scores of different emotions based on different spaces are obtained, thereby completing gait emotion recognition.

[0074] The principle of the present invention is:

[0075] The neural architecture search technology is used to automatically construct the backbone network for extracting three-dimensional human posture features in rectangular coordinate systems and spherical coordinate systems, and the emotion classification score is output through the Softmax classifier. The classification score is then input into the joint scoring module to output the individual emotional state and complete gait emotion recognition.

[0076] Embodiment 1:

[0077] like Figure 1 As shown, a gait emotion joint recognition method based on heterogeneous space neural architecture search specifically includes the following steps:

[0078] Step S1: Use the 3D human posture estimation algorithm VideoPose3D to extract the 3D human posture joints in different emotional video data, and the video data mainly contains happy, depressed and neutral emotions. The joints of the 3D human posture include root joints, spine joints, neck joints, head joints, left shoulder joints, left elbow joints, left hand joints, right shoulder joints, right elbow joints, right hand joints, left hip joints, left knee joints, left foot joints, right hip joints, right knee joints, right foot joints, etc.

[0079] Step S2: The extracted 3D human body posture joint points are spatially transformed into position information space and angle information space respectively; since the human body posture joint points extracted in step S1 belong to the rectangular coordinate system, it is only necessary to transform the 3D human body joint point coordinates in the rectangular coordinate system into the spherical coordinate system according to the spatial transformation formula. The rectangular coordinate system (x, y, z) and the spherical coordinate system The space transformation formula is as follows:

[0080]

[0081]

[0082]

[0083] Based on the similarity minimization strategy (using the cosine similarity calculation formula to calculate the feature similarity in the position information space mapping and the angle information space, and minimizing the similarity value during the training process, so that the features are mapped to different spaces), the joint points in different coordinate systems are mapped to the position information space and the angle information space.

[0084] Step S3: Construct neural architecture search space and joint search strategy in different spaces, use the joint search strategy to jointly search for the backbone network that extracts spatiotemporal emotion features in the position information space and angle information space, and complete the training based on the gait emotion dataset. The search space includes 3*3 convolution, 5*5 convolution, 3*3 maximum pooling, 3*3 average pooling, STGCN module, channel attention module, and spatial attention module; the joint search strategy is mainly composed of the joint gait emotion recognition accuracy, the backbone network feature extraction speed in the position information space, and the backbone network feature extraction speed in the angle information space, which can be specifically expressed as:

[0085]

[0086] Where ACC represents the joint gait emotion recognition accuracy, t c represents the backbone network feature extraction speed in the Cartesian position information space, T c represents the predefined feature extraction speed, t s represents the backbone network feature extraction speed in the angle information space, T s represents the predefined feature extraction speed, w1 and w1 are weight factors, which can be defined as:

[0087]

[0088]

[0089] Among them, α1, β1, α2, β2 are constants.

[0090] The optimal feature extraction backbone network in the position information space and angle information space is searched based on the joint search strategy. The specific search process is as follows: Figure 2 As shown, the details are as follows:

[0091] The controller first searches for the backbone network in the position information space and the angle information space in the search space, and then uses the gait emotion dataset to train and evaluate the model to obtain the joint gait emotion recognition accuracy and the feature extraction speed of the backbone network in each information space, and then feeds back to the controller according to the search strategy, prompting the controller to re-optimize the search for the backbone network in the position information space and the angle information space, and perform multiple iterations to obtain the optimal backbone network for extracting features in different spaces. Finally, the backbone network training is completed using the gait emotion dataset containing happy, depressed and neutral emotion labels.

[0092] Step S4: Use the backbone network trained in step S3 to extract the three-dimensional human body emotion features in the position information space and the angle information space respectively, and then use the Softmax classifier to perform emotion classification and scoring on the emotion features in different spaces.

[0093] Step S5: The different emotion scores in different input spaces are jointly scored by a joint scoring module, thereby completing gait emotion recognition. The joint scoring module can be expressed as follows:

[0094] F i =S i +C i

[0095] Where i represents the category, S i Indicates the scores of different sentiment categories in the angle information, C i represents the scores of different sentiment categories in the location information space, F i Represents the scores of different sentiment categories output by the joint scoring module.

[0096] Embodiment 2:

[0097] In one embodiment, the gait emotion joint recognition method based on heterogeneous space neural architecture search and the gait emotion dataset proposed in the present invention are used to search the backbone network in the information space and the angle information space, and the searched backbone network is trained. Using the trained backbone network, the temporal gait emotion features of the human body in different spaces are extracted and input into the Softmax classifier to obtain the scores of different emotions. Finally, the different emotion scores in different spaces are jointly scored through the joint scoring module, thereby completing gait emotion recognition. Figure 3 , which is a schematic diagram of the result of outputting the gait skeleton recognition output corresponding to the emotion of frustration in the current embodiment.

[0098] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of the method for joint recognition of gait and emotion based on heterogeneous spatial neural architecture search provided by the present invention and some or all steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0099] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0100] The present invention provides a method and idea for a gait emotion joint recognition method based on heterogeneous space neural architecture search. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A gait emotion joint recognition method based on heterogeneous neural architecture search, It is characterized in that The following steps are involved: Step 1, extracting 3D human body posture joint points in different emotion video data; Step 2, spatially transform the extracted 3D human body posture joint points into rectangular coordinate system and spherical coordinate system respectively, and perform position information space mapping and angle information space mapping on the joint points in different coordinate systems based on similarity minimization strategy; Step 3, constructing neural architecture search spaces and joint search strategies in different spaces, using the joint search strategy, respectively searching for optimal backbone networks in the neural architecture search spaces in different spaces, for extracting spatiotemporal emotion features in the position information space and the angle information space, and respectively training the above optimal backbone networks; Step 4: Use the trained optimal backbone network to extract the emotional features of human temporal gait in different spaces and input them into the Softmax classifier to obtain the scores of different emotions; Step 5: The different emotion scores in different input spaces are jointly scored through the joint scoring module to complete the joint recognition of gait emotions based on heterogeneous space neural architecture search.

2. According to claim 1, a gait emotion joint recognition method based on heterogeneous space neural architecture search, It is characterized in that The three-dimensional human posture joints described in step 1 include: root joints, spine joints, neck joints, head joints, left shoulder joints, left elbow joints, left hand joints, right shoulder joints, right elbow joints, right hand joints, left hip joints, left knee joints, left foot joints, right hip joints, right knee joints and right foot joints.

3. According to claim 1, a gait emotion joint recognition method based on heterogeneous space neural architecture search, It is characterized in that The spatial transformation described in step 2 specifically includes: Convert the three-dimensional human body posture joint points in the rectangular coordinate system extracted in 1 to the spherical coordinate system: Among them, any point in the rectangular coordinate system is (x, y, z), and the corresponding point in the transformed spherical coordinate system is 4. According to claim 1, a gait emotion joint recognition method based on heterogeneous space neural architecture search, It is characterized in that The optimal backbone network obtained in step 3 specifically includes the following steps: Step 3-1, constructing a neural architecture search space in different spaces, wherein the different spaces are: a position information space and an angle information space, and the neural architecture search space includes a submodule for constructing a backbone network; Step 3-2, construct a joint space search strategy, which is specifically expressed as follows: Among them, G represents the joint space search strategy, maximize represents maximization, ACC represents the joint gait emotion recognition accuracy, and t c represents the feature extraction speed of the backbone network constructed by each submodule in the neural architecture search space under the position information space, T c represents the predefined feature extraction speed, t s represents the feature extraction speed of the backbone network constructed by each submodule in the neural architecture search space under this angle information space, T s represents the predefined feature extraction speed, w1 and w2 are weight factors, as follows: Among them, α1, β1, α2, β2 are constants; Step 3-3, according to the joint search strategy constructed in step 3-2, search for the optimal combination of each sub-module in the neural architecture search space under the position information space and the angle information space as the optimal backbone network.

5. According to claim 4, a gait emotion joint recognition method based on heterogeneous space neural architecture search, It is characterized in that The submodules for constructing the backbone network described in step 3-1 specifically include: 3*3 convolution submodule, 5*5 convolution submodule, 3*3 maximum pooling submodule, 3*3 average pooling submodule, STGCN submodule, channel attention submodule and spatial attention submodule.

6. The method for joint gait and emotion recognition based on heterogeneous neural architecture search according to claim 5, It is characterized in that The optimal combination of each submodule in the neural architecture search space under the position information space and the angle information space described in step 3-3 is specifically searched as follows: Step 3-3-1, using a controller to search for the initial backbone network in the position information space and the angle information space respectively, wherein the backbone network is a random combination of submodules in the neural architecture search space; Step 3-3-2, using the gait emotion dataset to train and evaluate the above initial backbone network, and obtain the joint gait emotion recognition accuracy and the feature extraction speed of the initial backbone network in each space; Step 3-3-3, performing calculations according to the joint space search strategy described in step 3-2, and feeding the results back to the controller; Step 3-3-4, the controller re-optimizes the search for a new backbone network in the position information space and the angle information space, that is, re-randomly combines the submodules in the neural architecture search space to obtain a new backbone network; Step 3-3-5, using the new backbone network as the initial backbone network, repeatedly iteratively execute steps 3-3-2 to 3-3-4 until the preset iteration rounds, and obtain the optimal backbone networks in the position information space and the angle information space respectively.

7. The method for joint gait and emotion recognition based on heterogeneous neural architecture search according to claim 1, It is characterized in that The training of the optimal backbone network described in step 3 specifically includes: The optimal backbone network is trained using a gait emotion dataset containing emotion labels.

8. The method for joint gait and emotion recognition based on heterogeneous neural architecture search according to claim 1, It is characterized in that The scores of different emotions described in step 4 include: Step 4-1, using the optimal backbone network trained in step 3, extract the sentiment features in the position information space and the angle information space respectively; In step 4-2, the Softmax classifier is used to perform sentiment classification and scoring on the sentiment features in different spaces extracted in step 4-1 to obtain scores of different emotions.

9. The method for joint gait and emotion recognition based on heterogeneous neural architecture search according to claim 1, It is characterized in that The joint scoring described in step 5 specifically includes: The scores of different emotions obtained in step 4 are jointly scored through a joint scoring module to obtain joint scores of different emotions based on different spaces, and the emotion with the highest score is determined as the final result of the emotion recognition.

10. The method for joint gait and emotion recognition based on heterogeneous neural architecture search according to claim 9, It is characterized in that The joint scoring module described in step 5 is as follows: F i =S i +C i Among them, i represents the emotion category, S i represents the scores of different emotions in the angle information space, C i represents the scores of different emotions in the location information space, F i Represents the joint score output by the joint scoring module.

Images