Time phrase semantic analysis method and device in image search field

Abstract

The invention belongs to the technical field of image processing, and provides a time phrase semantic analysis method and device in the field of image search. The method comprises the following steps: preprocessing a search text in a to-be-processed image; based on a pre-training language large model BERT and a conditional random field CRF, establishing a sequence labeling model for identifying time expression in a context, obtaining a semantic feature vector corresponding to the to-be-processed image text, and inputting the obtained semantic feature vector into a time semantic analysis model to obtain a time identification result; and according to the obtained time identification result, carrying out subdivision category determination, and carrying out time fuzzification processing and result integration processing so as to return response information to the user. According to the method, the time type and the time information can be accurately identified, and more accurate search information can be responded to a user.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for semantic analysis of time phrases in the field of image search. Background Art

[0002] With the explosive growth of image data, traditional text-based search methods can no longer meet users' needs for image information retrieval. Intelligent image search technology uses computer vision, deep learning, and big data processing technologies to achieve efficient retrieval of image content, greatly improving the accuracy of image search and user experience.

[0003] At present, the relevant technologies of image search mainly involve the following fields: computer vision, image processing, machine learning and deep learning, information retrieval, big data processing, user interface and user experience. In addition, there is still a lot of room for improvement in many aspects such as how to improve the retrieval speed and obtain more accurate answer texts.

[0004] Therefore, it is necessary to provide a method and device for time phrase semantic analysis in the field of image search to solve the above problems. Summary of the invention

[0005] The present invention aims to provide a method and device for time phrase semantic analysis in the field of image search, so as to solve the technical problems in the prior art such as low accuracy and slow retrieval speed of image search, and poor user experience caused by the low accuracy and slow retrieval speed of image search. The technical problems to be solved by the present invention are achieved through the following technical solutions.

[0006] The first aspect of the present invention proposes a method for semantic analysis of time phrases in the field of image search, comprising: preprocessing the search text in the image to be processed; establishing a sequence annotation model based on a pre-trained language large model BERT and a conditional random field CRF, for identifying time expressions in context, specifically comprising: using the pre-trained language large model BERT to obtain a semantic feature vector of fuzzy time in the search text of the image to be processed; inputting the search text of the image to be processed into the sequence annotation model to obtain a semantic feature vector corresponding to the search text of the image to be processed, and inputting the obtained semantic feature vector into a time semantic analysis model to obtain a time recognition result; performing subdivision category determination based on the obtained time recognition result, performing time fuzzification processing and result integration processing; and returning response information to the user based on the result of the result integration processing.

[0007] The second aspect of the present invention proposes a time phrase semantic analysis device in the field of image search, which is used to execute the time phrase semantic analysis method in the field of image search described in the first aspect of the present invention. The time phrase semantic analysis device includes: a preprocessing module, which is used to preprocess the search text in the image to be processed; an establishment module, which establishes a sequence annotation model based on a pre-trained language large model BERT and a conditional random field CRF, for identifying time expressions in context, specifically including: using the pre-trained language large model BERT to obtain a semantic feature vector of fuzzy time in the search text of the image to be processed; an identification module, which is used to input the search text of the image to be processed into the sequence annotation model, obtain a semantic feature vector corresponding to the search text of the image to be processed, and input the obtained semantic feature vector into the time semantic analysis model to obtain a time recognition result; a determination module, which performs subdivision category determination according to the obtained time recognition result, performs time fuzzification processing and result integration processing; a response module, which returns response information to the user according to the result of the result integration processing.

[0008] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the time phrase semantic analysis method in the image search field described in the first aspect of the present invention.

[0009] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method for time phrase semantic analysis in the field of image search described in the first aspect of the present invention.

[0010] The embodiments of the present invention include the following advantages:

[0011] Compared with the prior art, the present invention pre-processes the search text in the image to be processed; based on the pre-trained language large model BERT and conditional random field CRF, a sequence annotation model is established to identify the time expression in the context, obtain the semantic feature vector corresponding to the image text to be processed, and input the obtained semantic feature vector into the time semantic analysis model to obtain the time recognition result; according to the obtained time recognition result, the subdivision category is determined, and the time fuzzification processing and result integration processing are performed to return the response information to the user. The present invention can accurately identify the time type and time information, and respond to more accurate search information for the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart of an example of a method for time phrase semantic analysis in the field of image search of the present invention;

[0013] Figure 2is a schematic diagram of an application example of the time phrase semantic analysis method in the field of image search of the present invention;

[0014] Figure 3 It is a structural block diagram of a time phrase semantic analysis system in the field of image search of the present invention;

[0015] Figure 4 is a schematic structural diagram of an electronic device embodiment according to the present invention;

[0016] Figure 5 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] In view of the above problems, the present invention proposes a method and system for semantic analysis of time phrases in the field of image search. The method pre-processes the search text in the image to be processed; based on the pre-trained language large model BERT and conditional random field CRF, a sequence annotation model is established to identify the time expression in the context, obtain the semantic feature vector corresponding to the image text to be processed, and input the obtained semantic feature vector into the time semantic analysis model to obtain the time recognition result; according to the obtained time recognition result, the subdivision category is determined, and the time fuzzification processing and result integration processing are performed to return the response information to the user. The present invention can accurately identify the time type and time information, and respond to more accurate search information for the user.

[0019] Example 1

[0020] Refer to the following Figure 1 , Figure 2 , the contents of the present invention will be described in detail.

[0021] Figure 1 It is a flowchart of an example of the method for time phrase semantic analysis in the field of image search of the present invention. Figure 2 It is a schematic diagram of an application example of the time phrase semantic analysis method in the field of image search of the present invention.

[0022] like Figure 1 As shown, in step S101, the search text in the image to be processed is preprocessed.

[0023] Specifically, the image to be processed is a screenshot obtained during the user's search process.

[0024] Furthermore, the image to be processed is, for example, a screenshot obtained during a user's search process for holidays. The image to be processed includes search text, for example, one or more keywords.

[0025] It should be noted that the search text is one word or multiple words, a sentence or a paragraph. The search text may be a search text containing keywords related to time, or may be a search text not containing keywords related to time.

[0026] Specifically, the keywords include words related to holidays and anniversaries. Specifically, they include words related to years, months, days, and quarters. Furthermore, they also include relative time, such as last year's Spring Festival, last year's summer vacation, last year's Children's Day, and last year's birthday.

[0027] For example, the following example:

[0028] BIOO

[0029] Last year at the beach.

[0030] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0031] Next, in step S102, based on the pre-trained language large model BERT and conditional random field CRF, a sequence labeling model is established to identify the time expression in the context of the search text, specifically including: using the pre-trained language large model BERT to obtain the semantic feature vector of fuzzy time in the search text of the image to be processed.

[0032] A sequence labeling model is established based on the pre-trained language model BERT and conditional random field CRF.

[0033] Specifically, historical search texts that users search in the cloud disk are collected, and the historical search texts contain text information with time semantics. Based on the three-digit sequence labeling method, the collected historical search texts with time semantics are labeled to obtain a training data set, where B represents the beginning of the time entity, I represents the middle or end of the time entity, and O represents the part that does not belong to the time entity.

[0034] The pre-trained language model BERT is used to extract the semantic feature vector of each search text sequence. The following expression is used to represent the semantic feature vector of each search text sequence:

[0035] h t =BERT(x) t

[0036] Among them, h tRepresents the implicit representation of the search text sequence corresponding to the t-th search text, specifically a feature vector of a specified dimension; x represents the search text in the image to be processed.

[0037] For the establishment of the sequence labeling model, the search text in the image to be processed is converted into a semantic feature vector, and after the implicit representation is determined, the label score of the image to be processed is calculated using the following expression:

[0038] s t =Wh t +b

[0039] Among them, s t represents the state score of the t-th text sequence corresponding to each time category; h t is the implicit expression of the text in the image to be processed; W is a weight matrix with a specified number of columns and a specified number of rows; b is a bias vector of a specific dimension.

[0040] Optionally, the specified number of rows of the weight matrix W is 2-10, and the specified number of columns of the weight matrix W is 500-800. The specific latitude of the bias vector b is 2-6.

[0041] Preferably, the specified number of rows of the weight matrix W is 3, the specified number of columns of the weight matrix W is 768, and the specific dimension of the bias vector b is 3.

[0042] Next, linear layer mapping is performed according to the extracted semantic feature vector of each search text sequence to obtain the label score space distribution to obtain a sequence labeling model.

[0043] In the conditional random field CRF layer of the sequence labeling model, the total score of the search text sequence corresponding to the search text of the image to be processed is calculated using the following expression according to the calculated state score:

[0044]

[0045] Among them, score(y,s) represents the total score of the text in the search text sequence corresponding to the search text, where s represents the state score sequence of all time categories corresponding to the search text, and y is the label sequence corresponding to the state score sequence; transition(y t-1 ,y t ) indicates that from y t-1 Transfer to y t Transition Score t Represents the state score of the t-th text sequence corresponding to each time category.

[0046] Using negative log likelihood loss, we calculate the loss value of the model training process:

[0047]

[0048] in, Represents the loss value of the model training process; score(y,s) represents the total score of the text of the image to be processed, where s represents the actual state score sequence of all time categories corresponding to the text of the image to be processed, and y is the label sequence corresponding to the state score sequence; score(y′,s) represents the state score sequence predicted by the model, s represents the actual state score sequence of all time categories corresponding to the text of the image to be processed, and y′ is the predicted label sequence corresponding to the state score sequence.

[0049] The pre-trained language model BERT is used to obtain the semantic feature vector of fuzzy time as the model input, which is input into the fully connected layer and the classification layer, and output as the classification result through the Softmax function:

[0050] Y = Softmax(Wz+b)

[0051] Among them, W is the weight matrix of the fully connected layer, z is the semantic feature vector output by BERT, b is the bias vector, and Y is the time category probability distribution;

[0052] During model training, the cross entropy loss function is selected to train the sequence labeling model. For each text sample, the goal is to minimize the model prediction value. and the true label value y c The difference:

[0053]

[0054] in, Represents the model prediction value and the true label value y c The difference of y c Represents the value of the true label (for example, one-hot encoding); It represents the probability distribution of the time type predicted by the model, that is, the model prediction value, C represents the number of categories, and C is a positive integer.

[0055] Thus, a trained sequence labeling model is obtained, which can identify time categories more accurately.

[0056] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0057] Next, in step S103, the search text of the image to be processed is input into the sequence labeling model to obtain a semantic feature vector corresponding to the search text of the image to be processed, and the obtained semantic feature vector is input into the temporal semantic analysis model to obtain a temporal recognition result.

[0058] In a specific embodiment, on the basis of the sequence labeling model, a classification layer based on probability distribution is added to obtain a temporal semantic analysis model, which is used to identify the following temporal content in the search text: nested time expressions, tense categories; the sequence labeling model outputs whether it contains time words, time entities and the number of time entities.

[0059] The temporal semantic analysis model outputs a time recognition result, which includes the recognized time category and a confidence or probability value corresponding to the time category.

[0060] For example, the recognition time category is the third type (ie, relative time type), such as "last year's National Day", with a confidence level of 0.7.

[0061] In another specific embodiment, the temporal semantic analysis model is a model independent of the sequence labeling model.

[0062] The first output result of the model of the sequence labeling model is used as the model input of the temporal semantic analysis model.

[0063] When the first output result includes a processing time anchor point, the first output result is input into a time semantic analysis model to obtain a time recognition result, wherein the time recognition result includes a time anchor point, an identified time category, and a confidence or probability value corresponding to the time category.

[0064] The time anchor point may be, for example, last year, this year, today, yesterday, the day before yesterday, anniversaries, birthdays, etc.

[0065] According to the semantic feature vector of the fuzzy time in the image to be processed, the time semantic analysis model is input to obtain the time recognition result.

[0066] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0067] Next, in step S104, the subdivision category is determined based on the obtained time recognition result, and time fuzzification processing and result integration processing are performed.

[0068] For time fuzzification processing, such as Figure 2 As shown, the specific steps include:

[0069] Step S201: Fuzzy time expression recognition, identifying time expressions containing uncertainty.

[0070] Indefinite time expressions such as "approximately", "around", "within a few days", etc.

[0071] Step S202: Distinguish different types of ambiguity.

[0072] The ambiguity includes quantity ambiguity (a few days), scope ambiguity (approximately), degree ambiguity (possibly), etc.

[0073] Step S203: semantic analysis, parsing the specific meaning of the fuzzy words and evaluating the degree of fuzziness.

[0074] For example, "approximately" may be more precise than "almost". Contextual information integration considers the additional information provided by the surrounding text and uses domain knowledge and common sense reasoning to supplement the information.

[0075] Step S204: Determination of a reference time point: for relatively ambiguous time, determine a reference time point.

[0076] For example, the reference time point is the current time, the document time, or another time point in the context.

[0077] For time range estimation, the possible time range is estimated based on the fuzzy expression, for example, "a few days later" indicates a range of 2-7 days.

[0078] Optionally, a temporal entity recognition algorithm (such as based on regular expressions, CRF, BERT model) is used to extract fuzzy time expressions, such as "recently", "not long ago", "a few months ago".

[0079] By extracting contextual information (such as related events, contextual prompt words) of fuzzy time expressions, the base time (such as "today", "last Friday") or reference event can be identified.

[0080] Step S205: Probability distribution modeling, creating a probability distribution model for fuzzy time.

[0081] Specifically, a normal distribution, a triangular distribution or other suitable probability distribution is used.

[0082] According to different fuzzy time expressions, the corresponding probability distribution model is established. For the choice of distribution,

[0083] Gaussian distribution is suitable for symmetric fuzzy statements, such as "a few days ago". Exponential distribution is suitable for decreasing statements, such as "the earlier the less certain". The central time point (μ\muμ) is defined by prior rules or training data, such as "5 days ago". The standard deviation (σ\sigmaσ) is used to define the degree of fuzziness, which can be adjusted through experience or model learning. The decay parameter (λ\lambdaλ) is used to control the time decay of the exponential distribution.

[0084] Optionally, the input natural language text is preprocessed, including word segmentation, part-of-speech tagging, dependency syntax analysis, etc.

[0085] Specifically, distribution parameters such as mean, standard deviation, minimum value, maximum value, etc. are set according to the degree of fuzziness. Precision level determination: Determine the time unit (year, month, day, hour, etc.) according to the precision of the expression, and adjust the output precision to match the fuzziness of the input. For highly ambiguous expressions, generate multiple possible interpretations and assign a probability or confidence level to each interpretation. Interval representation generation: Create a time interval representation, such as [earliest possible time, latest possible time], which may include confidence intervals, such as 90% confidence intervals. Fuzzy logic application: Use fuzzy logic to process imprecise values, define and apply fuzzy sets and fuzzy rules.

[0086] Step S206: Determine the fuzzy time type range and probability estimation according to the probability distribution model.

[0087] According to the probability distribution model established in the above steps, the fuzzy time type range and probability estimation are determined.

[0088] Specifically, the probability value of each specific date within the time range is calculated through the above probability distribution model.

[0089] Use the following expression to calculate the probability value of each specific date within the time range:

[0090]

[0091] Among them, P(t) represents the probability value of the current time t (i.e., the time corresponding to the current fuzzy time expression) on each specific date (specifically, at each fuzzy time); σ is the standard deviation calculated using each fuzzy time, which is used to characterize the degree of fuzziness of the fuzzy time. It can also be learned using an existing machine learning model or determined using empirical values ​​from historical data; μ represents the central time point of the Gaussian distribution.

[0092] Optionally, use the following expression to calculate the probability value of each specific date within the time range:

[0093] P(t)=λe -λt

[0094] Wherein, P(t) represents the probability value of the current time t (i.e., the time corresponding to the current fuzzy time expression) on each specific date (specifically, at each fuzzy time); λ represents the attenuation parameter, which is the parameter used for time attenuation control of the exponential distribution.

[0095] Preferably, it also includes dynamic context adjustment, specifically dynamically adjusting the distribution parameters according to the context information, wherein if a specific event (such as "yesterday", "Spring Festival") is mentioned in the context description, the reference time is adjusted and the distribution is recalculated.

[0096] In addition, based on contextual association models (such as BERT or GPT), the semantic dependency between temporal representation and reference points is strengthened.

[0097] In a specific implementation, the probability values ​​of each specific date (specifically corresponding to each fuzzy time) in each time range are determined by the calculated probability values. The probability values ​​of the fuzzy time expressions contained in the search text are calculated, and the time point with the highest probability is selected as the result of the search text.

[0098] If the result corresponds to a time range (such as "a few months ago"), the time range with the highest probability is output.

[0099] In addition, historical data analysis is also performed to analyze the actual situation of similar expressions in historical data, and statistical models are used to optimize time ranges and probability estimates. User preferences are considered. If there is user data, the understanding preferences of individuals or groups on fuzzy time are considered, and the time range and probability distribution are adjusted in a personalized manner.

[0100] Dynamically adjust the time granularity based on the degree of fuzziness, for example, "about a month later" does not need to be accurate to the day.

[0101] Preferably, the result representation is formatted to generate a standardized fuzzy time representation, which may include a central value, range, probability distribution parameters, etc. Interpretability information is generated to provide an explanatory description of the processing process and derive a specific time representation from the fuzzy expression. Result verification and adjustment: a rationality check is performed to ensure that the results are consistent with common sense and context, and manual review and adjustment are performed when necessary.

[0102] According to the above-mentioned time fuzzification processing, the following time fuzzification processing results are obtained: standardized fuzzy time representation of each fuzzy expression (including central value, range, probability distribution parameters, and optimized and adjusted time granularity, etc.), similar expressions of each fuzzy expression, and specific time representation obtained from each fuzzy expression.

[0103] Furthermore, the result integration processing is performed according to the obtained time fuzzy processing result.

[0104] The understanding preference of individuals or groups on fuzzy time is associated with the time fuzzification processing results through category analysis, such as cluster analysis, to obtain multiple categories of fuzzy time expressions related to user understanding preferences.

[0105] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0106] Next, in step S105, the result of the processing is integrated and response information is returned to the user.

[0107] According to the multiple types of fuzzy time expressions related to the user's understanding preference obtained in step S104, the relevant time fuzzy expression of the current user and the corresponding specific time expression are determined as response information and returned to the current user.

[0108] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0109] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0110] Compared with the prior art, the present invention pre-processes the search text in the image to be processed; based on the pre-trained language large model BERT and conditional random field CRF, a sequence annotation model is established to identify the time expression in the context, obtain the semantic feature vector corresponding to the image text to be processed, and input the obtained semantic feature vector into the time semantic analysis model to obtain the time recognition result; according to the obtained time recognition result, the subdivision category is determined, and the time fuzzification processing and result integration processing are performed to return the response information to the user. The present invention can accurately identify the time type and time information, and respond to more accurate search information for the user.

[0111] Example 2

[0112] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0113] Figure 3 4 is a schematic structural diagram of an example of a time phrase semantic analysis device in the field of image search according to the present invention.

[0114] Reference Figure 3The second aspect of the present disclosure provides a time phrase semantic analysis device 300, which adopts the time phrase semantic analysis method described in the first aspect of the present disclosure. The time phrase semantic analysis device in the field of image search includes a preprocessing module 310, a building module 320, a recognition module 330, a determination module 340, and a response module 350.

[0115] Specifically, the preprocessing module 310 is used to preprocess the search text in the image to be processed. The establishment module 320 establishes a sequence labeling model based on the pre-trained language large model BERT and the conditional random field CRF, which is used to identify the time expression in the context, specifically including: using the pre-trained language large model BERT to obtain the semantic feature vector of fuzzy time in the search text of the image to be processed. The recognition module 330 is used to input the search text of the image to be processed into the sequence labeling model, obtain the semantic feature vector corresponding to the search text of the image to be processed, and input the obtained semantic feature vector into the time semantic analysis model to obtain the time recognition result. The determination module 340 performs subdivision category determination based on the obtained time recognition result, and performs time fuzzification processing and result integration processing. The response module 350 returns response information to the user based on the result of the result integration processing.

[0116] According to an optional implementation manner, historical search texts searched by users in the cloud disk are collected, wherein the historical search texts contain text information with time semantics; and the collected historical search texts containing time semantics are annotated based on a three-digit sequence annotation method, wherein B represents the beginning of a time entity, I represents the middle or end of a time entity, and O represents a part that does not belong to a time entity.

[0117] The pre-trained language model BERT is used to extract the semantic feature vector of the search text sequence corresponding to each historical search text. The following expression is used to represent the semantic feature vector of each search text sequence:

[0118] h t =BERT(x) t

[0119] Among them, h t Represents the implicit representation of the t-th search text sequence, specifically a feature vector of the specified dimension; x represents the search text.

[0120] Linear layer mapping is performed according to the extracted semantic feature vector of each search text sequence to obtain the label score space distribution to obtain the sequence labeling model.

[0121] According to an optional implementation manner, the acquired semantic feature vector is input into a temporal semantic analysis model to obtain a temporal recognition result.

[0122] On the basis of the sequence labeling model, a classification layer based on probability distribution is added to obtain a temporal semantic analysis model, which is used to identify the following time categories in the text information in the image to be processed: nested time expressions, tense categories; the sequence labeling model outputs whether it contains time words, time entities and the number of time entities.

[0123] The temporal semantic analysis model outputs a time recognition result, which includes the recognized time category and a confidence or probability value corresponding to the time category.

[0124] According to an optional implementation, the acquired semantic feature vector is input into a temporal semantic analysis model to obtain a time recognition result, specifically including: the temporal semantic analysis model is a model independent of the sequence labeling model; the first output result of the model of the sequence labeling model is used as a model input of the temporal semantic analysis model; when the first output result includes a processing time anchor point, the first output result is input into the temporal semantic analysis model to obtain a time recognition result, the time recognition result includes the time anchor point, the recognized time category, and the confidence or probability value corresponding to the time category; according to the acquired semantic feature vector of fuzzy time in the search text of the image to be processed, the temporal semantic analysis model is input to obtain a time recognition result.

[0125] The establishment of the sequence labeling model includes: converting the search text in the image to be processed into a semantic feature vector, and determining the implicit representation,

[0126] The label score of the image to be processed is calculated using the following expression:

[0127] s t =Wh t +b

[0128] Among them, s t represents the state score of the t-th text sequence corresponding to each time category; h t is the implicit expression of the text in the image to be processed; W is a weight matrix with a specified number of columns and a specified number of rows; b is a bias vector of a specific dimension.

[0129] According to an optional implementation, in the CRF layer, based on the calculated state scores, the total score of the text of the image to be processed is calculated using the following expression:

[0130]

[0131] Among them, score(y,s) represents the total score of the search text of the image to be processed, where s represents the state score sequence of all time categories corresponding to the search text of the image to be processed, and y is the label sequence corresponding to the state score sequence; transition(y t-1 ,y t ) indicates that from y t-1 Transfer to y t The transfer score, i.e. Transition Score.

[0132] Using negative log likelihood loss, we calculate the loss value of the model training process:

[0133]

[0134] in, Represents the loss value of the model training process; score(y,s) represents the total score of the search text of the image to be processed, where s represents the actual state score sequence of all time categories corresponding to the search text of the image to be processed, and y is the label sequence corresponding to the state score sequence; score(y′,s) represents the state score sequence predicted by the model, s represents the actual state score sequence of all time categories corresponding to the search text of the image to be processed, and y′ is the predicted label sequence corresponding to the state score sequence.

[0135] According to an optional implementation, the pre-trained language model BERT is used to obtain the semantic feature vector of fuzzy time as the model input, which is input to the fully connected layer and the classification layer, and output as the classification result through the Softmax function:

[0136] y=Softmax(Wz+b)

[0137] Among them, W is the weight matrix of the fully connected layer, z is the semantic feature vector output by BERT, b is the bias vector, and y is the time category probability distribution.

[0138] During model training, the cross entropy loss function is selected to train the sequence labeling model. For each text sample, the goal is to minimize the predicted distribution value. and the true label value y c The difference:

[0139]

[0140] in, Represents the predicted distribution value and the true label value y c The difference of y c Represents the one-hot encoding of the true label, Represents the probability distribution of time types predicted by the model, C represents the number of categories, and C is a positive integer.

[0141] Compared with the prior art, the present invention pre-processes the search text in the image to be processed; based on the pre-trained language large model BERT and conditional random field CRF, a sequence annotation model is established to identify the time expression in the context, obtain the semantic feature vector corresponding to the image text to be processed, and input the obtained semantic feature vector into the time semantic analysis model to obtain the time recognition result; according to the obtained time recognition result, the subdivision category is determined, and the time fuzzification processing and result integration processing are performed to return the response information to the user. The present invention can accurately identify the time type and time information, and respond to more accurate search information for the user.

[0142] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0143] like Figure 4 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0144] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0145] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0146] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0147] It should be understood that Figure 4The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0148] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 5 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0149] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0150] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0151] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0152] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.

[0153] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0154] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0155] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A method for semantic analysis of time phrases in the field of image search, characterized in that: include: Preprocessing the search text in the image to be processed; Based on the pre-trained language model BERT and conditional random field CRF, a sequence annotation model is established to identify the time expression in the context, specifically including: using the pre-trained language model BERT to obtain the semantic feature vector of fuzzy time in the search text of the image to be processed; Inputting the search text of the image to be processed into the sequence labeling model, obtaining a semantic feature vector corresponding to the search text of the image to be processed, and inputting the obtained semantic feature vector into the temporal semantic analysis model to obtain a temporal recognition result; According to the obtained time recognition results, the subdivision categories are determined, and the time fuzzification processing and result integration processing are performed; The processing results are integrated according to the results, and the response information is returned to the user.

2. The method for semantic analysis of time phrases in the field of image search according to claim 1, characterized in that: The establishing of the sequence labeling model comprises: Collecting historical search texts of users in the cloud disk, wherein the historical search texts contain text information with time semantics; Based on the three-digit sequence annotation method, the collected historical search texts with time semantics are annotated, where B represents the beginning of the time entity, I represents the middle or end of the time entity, and O represents the part that does not belong to the time entity; The pre-trained language model BERT is used to extract the semantic feature vector of the search text sequence corresponding to each historical search text. The following expression is used to represent the semantic feature vector of each search text sequence: h t =BERT(x) t Among them, h t represents the implicit representation of the t-th search text sequence, specifically a feature vector of the specified dimension; x represents the search text; Linear layer mapping is performed according to the extracted semantic feature vector of each search text sequence to obtain the label score space distribution to obtain the sequence labeling model.

3. The method for semantic analysis of time phrases in the field of image search according to claim 1 or 2, characterized in that: The step of inputting the acquired semantic feature vector into a temporal semantic analysis model to obtain a temporal recognition result includes: On the basis of the sequence labeling model, a classification layer based on probability distribution is added to obtain a temporal semantic analysis model, wherein the temporal semantic analysis model is used to identify the following time categories in the text information in the image to be processed: nested time expressions, tense categories; the sequence labeling model outputs whether it contains time words, time entities and the number of time entities; The temporal semantic analysis model outputs a time recognition result, which includes the recognized time category and a confidence or probability value corresponding to the time category.

4. The method for semantic analysis of time phrases in the field of image search according to claim 1 or 2, characterized in that: The step of inputting the acquired semantic feature vector into a temporal semantic analysis model to obtain a temporal recognition result includes: The temporal semantic analysis model is a model independent of the sequence labeling model; Using the first output result of the model of the sequence labeling model as the model input of the temporal semantic analysis model; When the first output result includes a processing time anchor point, inputting the first output result into a time semantic analysis model to obtain a time recognition result, wherein the time recognition result includes the time anchor point, the recognized time category, and a confidence or probability value corresponding to the time category; According to the semantic feature vector of fuzzy time in the search text of the image to be processed, the time semantic analysis model is input to obtain the time recognition result.

5. The method for semantic analysis of time phrases in the field of image search according to claim 2, characterized in that: The establishing of the sequence labeling model comprises: After the search text in the image to be processed is converted into a semantic feature vector and the implicit representation is determined, the label score of the image to be processed is calculated using the following expression: s t =Wh t +b Among them, s t represents the state score of the t-th text sequence corresponding to each time category; h t is the implicit expression of the text in the image to be processed; W is a weight matrix with a specified number of columns and a specified number of rows; b is a bias vector of a specific dimension.

6. The method for semantic analysis of time phrases in the field of image search according to claim 4, characterized in that: include: In the CRF layer, based on the calculated state scores, the following expression is used to calculate the total score of the text of the image to be processed: Among them, score(y, s) represents the total score of the search text of the image to be processed, where s represents the state score sequence of all time categories corresponding to the search text of the image to be processed, and y is the label sequence corresponding to the state score sequence; transition(y t -1, yt) means from y t -1 is transferred to y t The transfer score, i.e. Transition Score.

7. The method for semantic analysis of time phrases in the field of image search according to claim 6, characterized in that: include: Using negative log-likelihood loss, we calculate the loss value of the model training process: in, Represents the loss value of the model training process; score(y, s) represents the total score of the search text of the image to be processed, where s represents the actual state score sequence of all time categories corresponding to the search text of the image to be processed, and y is the label sequence corresponding to the state score sequence; score(y′, s) represents the state score sequence predicted by the model, s represents the actual state score sequence of all time categories corresponding to the search text of the image to be processed, and y′ is the predicted label sequence corresponding to the state score sequence.

8. The method for semantic analysis of time phrases in the field of image search according to claim 7, characterized in that: include: The pre-trained language model BERT is used to obtain the semantic feature vector of fuzzy time as the model input, which is input into the fully connected layer and the classification layer, and output as the classification result through the Softmax function: y=S0ftmax(Wz+b) Among them, W is the weight matrix of the fully connected layer, z is the semantic feature vector output by BERT, b is the bias vector, and y is the time category probability distribution; During model training, the cross entropy loss function is selected to train the sequence labeling model. For each text sample, the goal is to minimize the predicted distribution value. and the true label value y c The difference: in, Represents the predicted distribution value and the true label value y c The difference of y c Represents the one-hot encoding of the true label, Represents the probability distribution of time types predicted by the model, C represents the number of categories, and C is a positive integer.

9. A device for analyzing time phrase semantics in the field of image search, characterized in that: It is used to execute the time phrase semantic analysis method in the field of image search according to any one of claims 1 to 8, and the time phrase semantic analysis device comprises: A preprocessing module, used for preprocessing the search text in the image to be processed; Establish a module, based on the pre-trained language model BERT and conditional random field CRF, to establish a sequence annotation model for identifying time expressions in context, specifically including: using the pre-trained language model BERT to obtain the semantic feature vector of fuzzy time in the search text of the image to be processed; A recognition module, used for inputting the search text of the image to be processed into the sequence labeling model, obtaining a semantic feature vector corresponding to the search text of the image to be processed, and inputting the obtained semantic feature vector into the temporal semantic analysis model to obtain a temporal recognition result; The determination module determines the subdivision category according to the obtained time recognition results, performs time fuzzification processing and result integration processing; The response module integrates the processing results and returns the response information to the user.

10. The device for time phrase semantic analysis in the field of image search according to claim 9, characterized in that: include: Collect text information containing time semantics, and annotate the collected historical search text containing time semantics based on the three-digit sequence annotation method, where B represents the beginning of the time entity, I represents the middle or end of the time entity, and O represents the part that does not belong to the time entity; The pre-trained language model BERT is used to extract the semantic feature vector of each search text sequence. The following expression is used to represent the semantic feature vector of each search text sequence: h t =BERT(x) t Among them, h t represents the implicit representation of the t-th search text sequence, specifically a feature vector of the specified dimension; x represents the search text; Linear layer mapping is performed according to the extracted semantic feature vector of each search text sequence to obtain the label score space distribution to obtain the sequence labeling model.

Images